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CHAPTER  I 

REVIEW  OF  THE  LITERATU'RE  ON  THE  APPLICATION 
OF  RATING  SCALES  IN  THE  EVALUATION 
OF  COLLEGE  TEACHING 

The  literature  reviewed  in  this  chapter  deals  with 
the  three  problems  which  were  encountered  early  in  this  in- 
vestigation, i.  e.,   (1)  the  qualities  which  are  most  essential 
in  a  successful  college  teacher;   (2)  the  type  of  measure  best 
adapted  to  the  evaluation  of  college  teaching  through  the  me- 
dium of  student  Judgment;  and  (3)  the  reliability  and  validity 
of  student  judgment  when  applied  to  the  evaluation  of  college 
teaching,  through  the  use  of  rating  scales.    All  the  studies 
which  are  reviewed  below  apply  to  college  teaching  and  college 
teachers,  and  only  those  parts  of  each  study  v;hich  have  a  close 
relation  to  the  problems  under  discussion  are  reviev/ed  in  de- 
tail. 

Studies  Relating  to  Qualities  of  College  Teachers 

Investigators  of  this  problem  have  made  use  of  the 
judgments  of  college  administrators,  college  teachers  and  col- 
lege students.    Among  those  based  upon  the  opinions  of  college 
teachers,  the  study  reported  by  F.  S.  Bree  dl  deserves  honorable 
mention,  because  of  the  careful  manner  in  which  it  was  made. 
Each  member  of  a  committee  appointed  for  the  purpose  submitted 
a  list  of  qualities  which  he  deemed  most  essential  to  success- 
or .  S.  Breed,  "A  Guide  for  College  Teaching",  School  and  So- 
ciety, XXIV  (July  17,  1925),  82-87. 
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ful  college  teaching.    These  lists  the  committee  worked  over 
and  a  new  list  was  developed  from  them.    This  new  list  was  sub- 
mitted to  thirty-one  experienced  college  teachers  for  sugges- 
tions and  corrections.    Upon  receiving  these  the  committee 
again  revised  the  list,  after  which  it  was  placed  before  a 
group  of  educational  experts.    The  following  are  the  chief 
items  of  the  final  list  arranged  in  the  order  of  importance  as 
determined  by  the  faculties  of  the  different  departments: 

I*  Knowledge  and  organization  of  subject 

matter.  1.6    Average  Rank 

a.  Possessing  a  broad  and  accurate  know- 
ledge of  the  subject  1 

b.  Selecting  the  material  of  the  course 
effectively  2 

c.  Organizing  the  course  so  that  se- 
quence of  the  topics  is  natural  and 

clear  3 

d.  Preserving  proper  balance  in  the 

emphasis  on  important  topics  4 

e.  Pointing  out  the  relationships  be- 
tween the  material  of  the  course  and 
other  subjects;  between  these  mater- 
ials and  current  affairs  5 

II,  Skill  in  Instruction  1.9 

a.  Getting  the  point  of  viev/  of  the 
student  and  adjusting  to  their 

powers  of  comprehension  1 

b.  Stimulating  intellectual  curiosity  2 

c.  Giving  evidence  that  the  daily  work 

is  carefully  planned  3 

d.  Making  clear  explanations  4 

e.  Conducting  discussions  with  skill  5 

f.  Helping  students  in  the  formation 

of  desirable  study  habits  6 

g.  Making  satisfactory  assignments  7 

h.  Returning  written  work  with 

criticisms  8 

III.  Personal  qualities  2.6 

a.  Interest  in  subject  1 

b.  Interest  in  teaching  2 
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c.  Sympathetic  attitude  toward  student, 
including  freedom  from  sarcasm  3 

d.  Tact  in  dealing  with  students  4 

e.  Open-mindedness  5 

f.  Sense  of  proportion,  including  a 

sense  of  humor  6 

g.  Accessibility  to  students  7 

h.  Self-reliance  and  confidence  8 

i.  Attractiveness  of  manner  9 
J.  Freedom  from  personal  idios3nicrasies 

that  interfere  with  effectiveness  10 


The  remaining  two  sections  of  the  list  are  not  included  in  this 
reviev/  as  they  do  not  include  factors  which  could  be  rated  by 
students. 

In  1930  Clinton^  reported  an  investigation  in  which 
177  students  were  asked  to  list  the  qualities  which  "they  de- 
sired in  their  college  teachers".    The  results  are  shown  in  the 
table  below. 

Quality  Number  of  students 


mentioning 

Knowledge  of  subject  matter  96 

A  pleasing  personality  73 

Neatness  in  appearance  and  work  57 

Fairness  to  all  students  50 

Kind  and  sympathetic  with  all  students  46 

A  keen  sense  of  humor  43 

Interest  in  the  profession  41 
Interesting  presentation  of  subject 

matter  41 

Alertness  and  broad-mindedness  36 

Knowledge  of  methods  34 

Understanding  of  human  nature  33 

Good  speaking  ability  33 

Good  character  and  reputation  32 

Reliable  and  honest  30 

Definiteness  of  requirements  29 


R.  J,  Clinton,  "Qualities  College  Students  Desire  in  College 
Instructors,"  School  and  Society,  XXXII ,  No.  8 30,( November  22, 
1930),  702. 
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Friendliness  29 

Tolerance  of  other  people ♦s  viev/s  29 

Enthusiasm  for  the  work  26 

Careful  organization  of  material  24 

Accurate  methods  of  grading  24 

A  good  social  mixer  21 

Cheerfulness  20 


Snavely'^,  in  an  address  reported  in  the  Association 
of  American  Colleges  Bulletin,  Volurae  XV,  March,  1929,  gives 
the  following  list  of  teacher  traits  which  were  determined  and 
arranged  by  a  committee  of  the  faculty  of  Birmingham-Southern 
College, 

FACTORS  IN  TEACHING 


I.  THE  INDIVIDUAL  FACTOR: 

a.  High  code  of  ethics  ) 

b.  Sincerity  ) 

c.  Grood  example 

d.  Good  breeding 

e.  Neat  appearance 

f .  Mastery  of  language 

g.  Poise  (absence  of  sarcasm) 

h.  Cheerfulness 

1.  Health  (Care  of) 


Integrity  of  character 


Culture 


II.  THE  SUBJECT  FACTOR: 

a.  Thorough  knowledge  of  special  subject  and  related 
fields 

b.  Knowledge  of  met:.ods  of  teaching  his  subject,  grad- 
ing, outside  reading,  exams,  lectures,  etc, 

c.  Enthusiasm 

1.  Willingness  to  work 

2.  Int&rest  in  subject 

3.  Eagerness  to  convey 

4.  Faith  in  the  value  of  the  subject 

5.  Originality 


III.  THE  STUDENT  FACTOR: 

a.  Appreciation  of  student's  viev/point  ) 

b.  Readiness  to  help  individuals  ) 


Sympathy  in 
classroom 


Guy  E.  Snavely,  "Who  Is  a  Great  Teacher",  Association  of  Amer- 
ican Colleges  Bulletin.  ZV,  No.  1,   (March,  1929),  68-72. 
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c.  Patience  ) 

d.  Fairness  ) 

e.  Interest  in  student  activities  )    Sympathy  outside 

f .  Interest  in  life  preparation      )       the  classroom 

Champlin^  has  reported  two  studies  which  were  de- 
signed to  determine  the  qualities  of  the  "Preferred  College 
Professor",  as  seen  by  the  students.  In  the  first  of  these 
the  following  sixteen  qualities  were  mentioned  most  often. 

1.  A  good  fellow  in  and  out  of  class 

2.  An  authority  in  the  knowledge  of  his  subject 

3.  A  congenial  companion 

4.  An  expert  in  the  work  of  teaching  others 

5.  Exceptional  ability  in  self-expression 

6.  Just,  impartial  and  sympathetic 

7.  Reasonable  always 

8.  Eager  to  assist  individual  students 

9.  Appreciation  of  the  student  viev^'point 

10.  Profound  understanding  of  human  nature  in  general 

11.  Possessing  a  good  name  in  the  community 
IS.  Making  a  good  appearance  in  public 

13.  Sincere  interest  in  the  personal  problems  of  students 

14.  Known  to  be  respectable  in  private  life 

15.  Tolerant  toward  the  opinions  of  colleagues  and  students 

16.  Capable  of  intellectual  growth 

In  his  second  study,  Champlin^  took  twenty  of  the 
thirty-five  traits  used  by  Clinton,  arranged  them  at  random  and 
submitted  them  to  275  students  who  were  asked  to  rank  them  in 
order  of  their  importance.    Below  is  the  li.'.'t  as  the  students 
arranged  them: 

1.  Thorough  knowledge  of  subject  matter 

2.  Interesting  presentation  of  subject  matter 

3.  Interest  in  the  profession 

4 

Carroll  D.  Charaplin,  "The  Preferred  College  professor,'*  School 
and  Society,  xmi  (February  11,  1928),  175-77. 
^Carroll  D.  Champlin,  "Attributes  Desired  in  College  Instruc- 
tors," School  and  Society.  XXXIII  (January  17,  1931),  17  5-77. 
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4.  Careful  organization  of  the  materials  of  instruction 

5.  Enthusiasm  for  professional  work 

6.  A  pleasing  personality 

7.  Fairness  to  all  students 

8.  Understanding  of  huiaan  nature 

9.  Knowledge  of  methods 

10.  Alertness  and  broad-mindedness 

11.  Good  character  and  reputation 

12.  Good  speaking  ability 

13.  Tolerance  of  other  people's  view 

14.  Accurate  methods  of  grading 

15.  Kindness  and  sympathy  for  all  students 

16.  Definiteness  of  requirements 

17.  A  keen  sense  of  humor 

18.  Reliability  and  honesty 

19.  Neatness  in  personal  appearance  and  work 

20.  Friendliness 


An  inspection  of  these  two  lists  reveals  a  consider- 


able amount  of  agreement  not  only  in  the  choice  of  the  traits 
but  also  in  the  relative  importance  attached  to  each  one.  The 
first  list  was  compiled  by  students,  while  the  second  was  com- 
piled by  faculty  members  but  arranged  by  student  preference. 


dents  to  list  the  elements  which,  to  them,  made  a  teacher  reaUy 
successful,  gave  the  following  arrangement: 


A  study  by  Davis^,  in  which  he  asked  77  college  stu 


Rank 


Quality 


Number  of 
students 
mentioning 
trait 


1 


2 


3 
4 


Personal  interest  in  students — sympathy 

and  friendliness 
Teaching  ability — ability  to  present 

subject  interestingly  and  to  inspire 

and  stimulate 
Personality 

General  interest  and  enthusiasm  for 
teaching  and  for  subject  matter 


76 


58 
48 


43 


C.  0.  Davis,  "Our  Best  Teachers,"  School  Review,  XXXIV  (Decem- 
ber, 1926),  754-59. 
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5 

neat  and  attractive  in  appearance 

41 

5 

Masterful  knowledge  of  subject  taught 

26 

7 

General  knowledge  or  culture 

24 

8 

knowledge  of  students 

22 

9 

Sociable,  numan,  friendly 

23 

10 

rower  to  discipline 

18 

11 

dheerful  and  even-tempered 

16 

12 

j^'aitii  and  conl'idence  in  pupils 

lb 

12 

ileen  sense  of  Jaumor 

lb 

14 

Willingness  to  iielp  otJaers  and  to 

cooperate 

lb 

lb 

i'air  but  exacting  in  work  assigned 

14 

16 

A  pleasing  voice 

13 

17 

Systematic  in  preparation 

11 

There  were,  in  all,  thirty-five  characteristics  sug- 
gested by  the  students,     in  the  foregoing  list  those  character- 
istics receiving  less  than  eleven  votes  were  omitted. 

Another  investigator,  jjrances  l.  Copper''',  had  49nstu- 
dents  in  a  teacher  training  class  in  northern  State  Teachers 
College  at  Marquette,  iviichigan,  list  the  traits  they  considered 
most  important  in  teachers.     The  ones  v/hich  were  mentioned  a 
significant  nuiiiber  of  ti:jes  are  as  follows: 


Personality 

22  times 

Willingness  to  help  students 

18 

Patience 

14 

impartiality 

12 

Discipline 

11 

interesting 

V 

Appearance 

6 

luiowledge  of  subject  inatter 

6 

Cheerfulness 

b 

interest  in  pupils 

b 

A  rather  extensive  study  of 

the  same  sort  is  reported 

by  i?'.  P.  u'Brien°.     rhis  study  involved  9b7  students  in  2b  of 

'^rrances  LeKoy  uopper,  "Who  is  a  good  Teacher?*^  EducationT' 
XnOLTK  (September,  1928j,  111-17. 
i)".  P.  O'Brien,  V/hat  Students  Say  About  college  instruction  and 
instructors ,  university  of  Kans&s  Bulletin,  vol.  XI  no.  8. 
Lawrence,  Kansas,     university  of  nansas,  1920.    pp.  20-26. 
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the  departments  in  the  University  of  Kansas,    Each  student  was 
asked  to  check,  on  a  given  list,  the  qualities  which  marked 
"the  best  college  teacher"  and  the  "feature  of  his  course 
which  caused  the  latter  to  meet  with  marked  approval,"  The 
traits  designated,  with  the  percentages  of  responses  assigned 
to  each,  were  as  follows: 

1.  He  seemed  to  have  more  complete  knowledge  of  his 

subject  than  have  most  teachers  22 ,4 

2.  His  personality  impressed  me  20,8 

3.  He  organized  his  course  so  that  it  invited 

initiative  and  encouraged  originality  13,4 

4.  He  appreciated  individual  needs  and  gave 

individual  encouragement  13.4 

5.  He  emphasized  concrete  applications  of  the 

content  of  the  course  11.0 

6.  He  organized  the  course  so  that  it  could  be 

grasped  v/ith  less  work  than  the  average  course  11.0 

7.  He  was  companionable  with  students  7.4 

In  a  more  detailed  analysis  of  the  personality  factor 
the  students  were  asked  to  check  four  of  a  list  of  twelve 
traits.    The  percentage  of  total  responses  for  each  of  the 
characteristics  named  is  reported  as  follows: 


Personality  Traits 

Percentage 

1. 

Intellectual  keenness 

13.4 

2. 

Intense  interest  in  subject 

12.9 

3, 

Sense  of  humor 

12.5 

4. 

Fairness  of  attitude 

12.4 

5. 

Approachableness 

8.0 

5. 

Cultured  in  manner 

7.1 

7. 

Taried  interests 

6.8 

8. 

Personally  efficient 

6.5 

9. 

Pleasing  voice 

6.3 

10. 

Naturalness 

5.1 

11. 

Well-groomed 

4.5 

12. 

Impressive  manner 

4.3 

In  the  studies  which  we  have  reviewed  it  will  be 


noted  that  some  investigators  made  use  of  student  judgment 
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while  others  secured  the  opinion  of  faculty  members.  Numerous 
expressions  of  opinion  as  to  the  qualities  required  in  the  suc- 
cessful college  teacher  have  been  made  by  college  adiainistra- 
tors  but  no  attempt  has  been  made  to  include  these  in  this  re- 
view since  they  must  be  regarded  as  being  even  less  scientific- 
ally determined  than  those  which  have  been  reviewed  above. 

Brooks^  reports  a  study  in  which  thirty- two  Rhodes 
scholars  were  asked  **to  recall  the  teacher  of  whom  you  can  say: 
'Ke  (or  she)  is  the  best  college  teacher  I  ever  had,*"  They  were 
then  asked  to  check  any  of  a  list  of  statements  which  they  be- 
lieved to  be  true  of  the  teacher  v/hom  they  regarded  as  their 
best  collewge  teacher. 

The  bulk  of  the  replies  were  separated  into  two  groups: 
(1)  Personal  qualities,  and  (2)  classroom  method.    Brooks  quotes 
the  follov/ing  representative  comments: 
"(1)   ♦Perfectly  human,* 

*A  keen  sense  of  humor;  interest  in  all  student  activities.* 

♦Energetic,  enthusiastic,  inspired  thought  and  action.* 

•Personal  charm.* 

♦A  remarkable  personality,* 
(2)  *His  position  in  the  class-room  was...  that  of  a  direc- 
tor of  discussion— not  that  of  a  tyrant,* 

*A  contagious  enthusiasm  and  love  for  his  subject,* 

*Always  encouraging  his  pupils,* 

The  able  lecturer  was  extolled  by  some,     *Brilliant  lec- 
turer.   Voted  by  each  succeeding  senior  class   *s  most 

inspiring  instructor, * 

*A  most  remarkable  flow  of  language  and  a  gift  for  using 
apt  illustrations,* 

*He  so  related  his  ideas  that  at  the  end  one  could  see  some- 
thing definite  had  been  established.* 

The  leader  of  men  was  the  ideal  of  many, 
♦He  was  the  hardest  worker  on  the  faculty.     One  learned  from 


W.  S.  Brooks.  "The  Rhodes  Scholars*  Ideal  Professor,"  School 
and  Society,  XXI  (March  28,  1925),  375-77. 
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him  how  a  real  student  should  work,* 

♦Ke  is  intensely  interested  in  his  work,  making  his  phil- 
osophy a  religion,  himself  a  preacher,* 
♦Interested  in  men  rather  than  in  events,*" 

The  question  quite  naturally  arises  as  to  which  groi^), 
college  teachers  or  college  students,  is  the  more  capable  to 
distinguish  the  traits  most  essential  in  high  grade  college 
teaching.    Breed^^  reports  a  study  which  shows  very  close  agree- 
ment between  the  two  groups  and  suggests  that,  in  view  of  the 
small  difference  discovered,  the  foregoing  question  need  not  be 
considered.     In  this  study,  fifty-six  faculty  members  and  one 
hundred  college  students  were  asked  to  rank  in  the  order  of 
their  importance  the  thirty-four  carefully  selected  instruction- 
al activities  and  qualities  grouped  under  five  large  headings. 
He  finds  no  significant  difference  between  the  ranking  by  the 
students  and  by  the  faculty  members.    In  fact,  the  ranking  is 
practically  identical  as  shown  in  the  following  table. 

Average  Ranking 


Faculty  Student 

I,  Knowledge  and  organization  of  sub- 
ject matter  1.6  1.7 
II,  Skill  in  instruction  1.9  1,9 
III.  Personal  qualities  2,5  2,7 
lY.  Professional  development  4,0  4.0 
Y.  University  cooperation  4,9  4,7 


It  is  interesting  to  note  that  personal  qualities 
vrere  not  placed  at  the  top  by  either  the  teachers  or  students. 
One  of  the  conmion  criticisms  of  rating  scales  is  that  the  per- 


Z.  S.  Breed,  **Factors  Contributing  to  Success  in  College 
Teaching,**  Journal  of  Educational  Research,  XYI  (November, 
1927),  247-53. 


-11- 

sonality  of  the  person  being  rated  tends  to  overshadow  every- 
thing else  and  to  create  what  is  termed  a  "halo  effect."  Not 
only  was    "-he  ranking  on  these  five  main  divisions  of  the  list 
practically  identical,  hut  on  the  smaller  items  into  which  each 
of  the  five  was    sub-divided  the  agreement  between  the  rankings 
given  by  students  and  those  given  by  the  faculty  members  was 
quite  marked. 

In  this  same  study  Breed'^''"  attempted  to  ascertain  the 
reliability  of  the  students*  ranking  of  the  items  in  the  list. 
The  hundred  students  were  divided  into  two  groups  of  fifty  each 
and  the  rank  values  for  each  of  the  thirty-four  items  were  com- 
puted separately  for  these  groups,  v/ith  the  following  results. 

There  was  perfect  agreement  on  19  items 

There  Y/as  a  difference  of  one  place  in  rank  on  12  items 
There  was  a  difference  of  two  places  in  rank  on         3  items 

The  selection  of  traits  to  be  used  in  a  rating  scale 
is  a  matter  of  prime  importance.    All  the  desirable  character- 
istics of  a  successful  teacher  are  not  of  such  a  nature  that 
they  can  be  observed  and  evaluated  very  accurately  by  students. 
The  difficulty  of  choosing  the  traits  in  a  scale  is  accentuated 
by  the  necessity  to  restrict  the  numbers  of  items  to  a  rela- 
tively few.     This  makes  it  very  easy  for  a  critic  of  any  spe- 
cific rating  scale  to  complain  that  some  very  essential  and 
desirable  trait  has  been  excluded. 

Tlie  task  of  selecting  the  items  in  a  rating  scale  and 
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the  details  of  the  construction  of  the  scale  have  been  made 
soraewhat  less  difficult  by  the  formulation  of  some  very  defi- 
nite standards  and  suggestions  by  experienced  scientific  vrork- 
ers  in  this  field, 

Brandenburg  and  Remmers-^^  suggest  the  following  as 
important  principles  in  the  selection  of  traits  for  a  scale  to 
be  used  by  students: 

(1)  The  list  must  be  relatively  short  so  that  the  student 
shall  not  be  lost  in  a  maze  of  different  qualities, 
(Halo  Effect), 

(2)  The  traits  must  be  such  as  are  generally  agreed  upon 
by  competent  critics  as  most  important  or  at  least  they 
must  be  among  the  most  important, 

(3)  The  traits  must  be  of  such  a  nature  that  they  are  fair- 
ly susceptible  to  student  observation  and  judgment, 

Patterson-'-^  claims  that  the  following  principles  are 

generally  accepted,  in  rating  human  qualities: 

(1)  Estimates  should  be  based  on  qualities  which  are  de- 
fined unambiguously  in  advance. 

(2)  Qualities  to  be  rated  should  be  defined  in  objective 
terms  as  far  as  possible. 

(3)  Each  quality  to  be  rated  should  refer  to  one  type  of 
activity  carried  on  or  to  one  trype  of  result  achieved 
by  those  to  be  rated, 

(4)  Rating  should  be  confined  to  past  and  present  accom- 
plishments. 

(5)  The  list  of  qualities  to  be  rated  must  be  related 
directly  to  the  type  of  v/ork  performed  by  those  to  be 
rated. 

(6)  The  method  of  recording  one's  ratings  should  be  easily 
understood  and  complied  with, 

(7)  Estimates  should  be  expressed  in  a  uniform  manner  by 
all  raters, 

(8)  As  many  judges  as  possible  should  be  employed  in  rating 

___ 

G.  G.  Brandenburg  and  H,  H.  Remiaers,  "Rating  Scales  for  In- 
structors", Educational  Administration  and  Supervision,  XIII 
(September,  1927),  399-405. 

D.  G.  Patterson,  "Methods  of  Rating  H\3man  Qualities,"  Annals 
of  the  American  Academy  of  Political  and  Social  Science,  CX 
(1923),  81-93. 
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a  given  person  and  an  average  of  all  available  ratings 
should  be  used  as  an  index  for  that  person. 

Although  there  were  twelve  principles  in  all,  the 
last  four  have  been  omitted  as  they  had  no  close  relation  to 
the  problem  of  rating  teachers. 

Another  experienced  worker  in  the  field  of  human 
measurement,  Bruce  Moore^^,  describes  extensive  experiments 
carried  on  in  connection  with  the  placement  of  300  graduate 
engineers  taken  on  each  year  by  the  Westinghouse  Electric  and 
Manufacturing  Company  and  sets  up  six  standards  or  principles 
which  follow. 

(1)  The  traits  selected  should  be  the  most  irr.portant  for  • 
success  in  that  particular  situation. 

(2)  The  different  traits  determined  upon  must  be  really 
different  and  as  distinct  from  each  other  as  possible. 

(3)  The  person  doing  the  rating  must  be  well  acquainted 
with  the  one  to  be  rated. 

(4)  The  trait  must  be  as  sharply  defined  as  possible  so 
that  all  raters  will  rate  the  same  trait. 

(5)  For  rating  any  individual  on  a  trait  the  basis  of  com- 
parison should  be  as  concrete  and  familiar  as  possible, 

(6)  Ratings  should  be  made  by  more  than  one  person. 

A  careful  reading  of  the  literature  concerning  the 
personal  and  professional  qualities  requisite  to  successful 
teaching  and  the  evaluation  of  the  same  leads  one  to  the  be- 
lief that  this  is  still  very  largely  a  problem  of  a  subjective 
nature  and  one  not  amenable  to  strictly  objective  scientific 
treatment,  at  least,  with  our  existing  techniques.  However, 
it  would  seem  to  be  the  part  of  wisdom  to  make  some  use  of  the 


"Bruce  Moore,  "Personnel  iSelection  of  Graduate  ilngineers , " 
Psychological  Monograph.  XXX  (1921),  84-90. 
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studies  which  have  been  reviewed,  in  the  construction  of  a 
scale  on  which  to  experiment,  if  for  nothing  more  than  as  a 
check  upon  the  judgment  of  the  experimenter. 

It  was  in  some  such  v/ay  that  the  follovdng  items 
v/ere  decided  upon  and  incorporated  into  the  College  Teachers 
Rating  Scale,  in  its  original  form.     This  form  was  changed  only 
slightly  in  the  final  revision.   (See  Appendix  A,  Exhibits  1  and 
2.) 

I,  Preparation  for  class  meetings 
II.  Interest  and  enthusiasm  in  his  subject 
III.  Ability  to  arouse  interest  in  his  subject 
IV.  Feeling  between  instructor  and  students 

Y.  Organization  of  course 
YI.  Thinking  demanded  of  students 
VII.  Ability  to  express  thought 
VIII.  Enunciation 

IX.  Leading  discussion  and  questioning 
X.  Personal  appearance 
XI.  Personal  peculiarities 
XII.  Interest  in  students 
XIII.  Sense  of  proportion 
XIV.  Sense  of  humor 

XV.  Self-confidence 
XVI.  Tolerance  and  liberality. 


Studies  Relating  to  the  Types  of  Measuring  Devices 

Scientific  data  concerning  the  relative  effectiveness 
of  the  different  varieties  of  devices,  designed  to  measure  the 
quality  of  instruction  in  colleges,  are  very  scanty.  Q^uestion- 
naires,  checking  lists,  rankings  and  rating  scales  have  all 
been  used  or  suggested  for  this  purpose  and  each  type  has  its 
advocates.    Probably,  also,  each  one  possesses  certain  advan- 
tages over  the  others  but  there  is  very  little  conclusive  ex- 
perimental evidence  concerning  their  comparative  superiority. 
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Guthrie-^^,  in  what  v/ould  seen  to  "be  a  very  carefully 
conducted  study,  compared  the  ranking  and  the  rating  scale 
methods  and  found  the  latter  to  be  about  fifteen  per  cent  more 
reliable.    The  rating  scale  which  he  used  is  similar  in  con- 
struction to  the  one  developed  in  this  study,  being  of  the  type 
comraonly  khown  as  a  "graphic  rating  scale."    There  are  other 
types  of  rating  scales  in  use  but  one  searches  in  vain  for  any 
convincing  experimental  evidence  as  to  their  comparative  merits, 
Hov/ever,  certain  investigators  in  rating  have  developed  some 
rather  decided  preferences  for  the  graphic  type  as  a  result  of 
their  work  vdth  the  different  types.    Freyd-^^  submits  the  fol- 
lowing general  advantages  of  the  graphic  rating  scale: 

1.  It  is  simply  and  easily  grasped, 

2.  It  is  interesting  and  requireiS  little  motivation  of  the 
rater, 

3.  It  is  quickly  filled  out, 

4.  It  is  simply  and  easily  scored, 

5.  It  frees  the  raters  from  direct  quantitative  terms. 

6.  It  enables  the  rater  to  make  his  discriminations  as 
fine  as  he  cares  to, 

7.  The  fineness  of  the  scoring  method  may  be  altered  at 
will,  yielding  scores  of  from  one  to  five  or  from  one 
to  one  hundred, 

8.  It  allov/s  of  comparable  ratings  without  requiring  each 
rater  to  know  all  the  members  of  the  group, 

Kornhauser"^    attributes  to  the  graphic  rating  scale 

the  following  advantages: 

1,  It  is  analytic;  it  calls  for  judgments  on  a  variety  of 

R.  Guthrie,  "Measuring  Students »  Opinions  of  Teachers," 
School  and  Society,  XXV  (February  5,  1987),  175-76. 

•^°I.Iax  Freyd,  "A  Graphic  Rating  Scale,"  Journal  of  Educational 
Psycholog^y.  XIY  (February,  1923),  82-102. 

■'■"Arthur  Y/.  Kornhauser,  "Ti/hat  Are  Rating  Scales  Good  For?" 
Journal  of  Personnel  Research,  V  (September,  1926),  189-0.3, 


distinct  and  defined  character  traits. 

2.  The  ratings  are  explicit  and  unambiguous  and  they  are 
recorded  so  that  they  may  be  cneckea  over,  compared  and 
otherwise  studied  after  the  rating  is  made, 

3.  The  ratings  are  uniform  and  standardized. 

4.  The  ratings  are  quantitative. 

5.  Ratings  may  be  given  periodically  and  compared. 

6.  The  graphic  rating  scale  is  extremely  easy  to  use. 

Hayes  and  Paterson^S  report  finding  the  graphic  rat- 
ing method  highly  reliable,  as  shown  by  the  close  relationship 
between  ratings  on  the  same  men  by  the  same  Judges  for  differ- 
ent months  and  by  a  close  relationship  between  ratings  on  the 
same  men  by  different  judges.    Symonds      also  expresses  his  be- 
lief that  the  graphic  rating  scale  is  the  best  form  to  use. 

20 

Watson    , in  listing  the  findings, regarding  rating 

scales,  which  have  been  established  more  or  less  firml^r  through 

the  experimental  work  of  the  last  twenty  years,  has  this  to  say 

for  the  graphic  type  of  scale • 

"The  graphic  rating  scale  in  which  the  rater  places  a 
check  upon  a  line  rather  than  using  statistical  terms 
has  advantages  in  permitting  fine  discriminations  and 
in  being  congenial  to  raters.'* 

21 

Marsh  and  Perrin     ,  on  the  other  hand,  claim  that 

■^^I.  H.  S.  Hayes,  and  D.  J".  Paterson,  "Experimental  Development 
of  the  Graphic  Rating  Method,"  Psychological  Bulletin,  XVIII 

-^g  (1921),  98-99. 

P.  M.  Symonds,  "ilotes  on  Ratings,"  Journal  of  Applied  Psy- 
c^-ology»  ^  (1925),  188-96. 

^^G.  B.  vVatson,  "Supplementary  Review  of  Measures  of  Personal- 
ity Traits,"  Journal  of  Educational  Psychology,  XVIII  (Feb- 
ruary,  1927),  73. 

'^-'-Sarah  E.  Marsh  and  P.  A.  0.  Perrin,  "An  Experimental  Study  in 
Rating  Scale  Technique,"  Journal  of  Abnormal  and  Social  Psy- 
chology, XIX  (March,  1915),  383-99. 
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their  experiments  fail  to  demonstrate  the  superiority  of  any- 
one type  of  scale.  They  also  report  practically  no  slp/n  of 
"halo  effect",  which  some  writers  claim  is  an  invariable  and 

unavoidsble  weakness  in  any  rating;  scheme  for  measuring  human 

22 

traits.     Syraonds      claims  that  ra.ting  and  rankin;;;^  are  equally 

reliable  but  disagrees  with  Marsh  and  Perrin  in  finding  the 

23 

"halo  effect"  to  be  a  reality.    Remmers  and  Brandenburg  also 
report  the  "halo  effect"  as  being  rather  insignificant,  but 
Webb^  ,  Thorndike    ,  Knight  and  Franzen  ^  have  reported  its 
existence  in  more  than  merely  significant  quantities  in 
their  studies.     It  is  possible  that  the  amount  of  "halo  effect" 
is  a  function  of  the  particular  scale  rather  than  a  necessary 
feature  of  all  rating  scales.    This  writer  has  failed  to 
find  any  convincing  experimental  evidence  on  this  Question, 
but  found  no  dearth  of  opinion  concerning  its  existence. 

Another  t)oint  in  the  construction  of  a  graphic  rating 
scale  involves  the  question  as  to  wa'-?ther  the  high  and  low  ends 
of  the  lines  on  the  scale  should  be  altern^ited  or  in  som.e  way 
staggered  so  that  all  the  lov/  or  all  the  high  ends  of  the  lines 

^^F.  M.  Symonds,  Otd.  cit,  d.  189. 

H.  H.  Remmers  and  Gr.  C.  Brandenburg,  "Experimental  Data  on  the 
Purdue  Rating  Scale  for  Instri-ctors , "  EciucationDl  Adminietra- 
tion  and  Supervision,  XIII  {llovemher,  1927},  519-27. 
24   ^  '  *  * 

£•  Webb,  "Character  and  Intelligence,"  British  Journal  of 
Psychological  Monog:ra-ohs ,  No.  3,  1915,  1-99. 

OR     '  '      '  ' 

"E.  L.  Thorndike,  "A  Constant  Error  in  Psychological  Ratings," 
Journal  of  Applied  PsvcholOr'y,  IV  (March,  1920),  25-29. 

^^F.  B.  Knight  and  R.  H.  Franzen,  "Pitfalls  in  Rating  Schemes," 
Journal  of  Educational  Psycnolo^y,  XIII  (April,  1922),  20^ 
13. 
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are  not  at  the  same  side  of  the  sheet.    Kemmers  and  j^randen- 
27 

burg      experimented  with  this  problem  in  constructing  their 
scale  and  found  no  difference  between  the  staggered  and  the 
straight  forms.    Conclusive  evidence  of  the  existence  of  any 
significant  difference  between  the  relative  effectiveness  of 
the  two  forms  seems  to  be  lacking,    it  may  well  be  that  this  is 
also  a  function  of  the  specific  scale  rather  than  a  general 
weakness  or  phenomenon. 

After  a  careful  consideration  of  the  different  types 
of  scales  it  was  decided  to  adopt  the  "graphic"  form  for  the 
College  Teachers  Hating  Scale ♦ 

The  Reliability  and  validity  of  Student  Judgmait 
When  Applied  to  the  Evaluation  of  Teachers  by 
the  use  of  Hating  Scales 

The  term  "reliability,"  when  used  in  describing  de- 
vices for  measuring  human  qualities,  refers  to  the  similarity 
or  coincidence  between  measures  or  ratings  made  of  the  same  per- 
son at  different  times  by  the  same  judges  or  raters.     Such  con- 
sistency or  similarity  is  usually  expressed  in  terms  of  the  co~ 

28 

efficient  of  correlation    between  the  ratings  made  at  different 

29 

times,  although  Freyd      claims  that  agreement  between  judges  is 

a  more  valid  criterion  of  reliability  than  consistency  of  indi- 

30 

vidual  judgments,    "Validity"     ,  on  the  other  hand,  refers  to 

27  — ' 
pJH.  H.  Hemmers  and  G.  C.  Brandenburg,  Op.  cit.,  p.  520. 

Tienry  E.  Garrett,  Statistics  in  Psychology  and  EducaU  oi ,  New 
_     York:  Longmans,  Green  and  Co.,  1926.  p.  268. 
^^Max  Freyd,  Op.  cit.  p.  90. 
^^Henry  E.  Garrett,  Op.  cit,  p.  266. 
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the  extent  to  whicn  the  measuring  device  measures  the  thing  it 
is  supposed  or  designed  to  measure.    Such  are  the  meanings  at- 
tached to  the  two  terms  in  this  study,    it  will  be  noted  that 
the  question  or  reliability  refers  not  only  to  the  judgments  of 
students  concerning  the  quality  of  teaching  in  general  but  also 
to  the  reliability  of  the  particular  scale  on  which  the  Judg- 
ments are  recorded,     if  one  scale  is  found  to  enable  students 
to  make  more  reliable  judgments  than  can  be  made  by  the  use  of 
a  second,  then  we  can  say  that  the  first  scale  is  more  reliable 
than  the  second. 

The  reliability  of  ratings  has  been  studied  by  a  great 
number  of  students  of  the  problem  with  varying  results.  Kela- 
tively  few  or  the  studies  deal  with  the  rating  of  college  teach- 
ers by  college  students  and  we  shall  limit  our  review  to  these 
few. 

Guthrie*'-^  reports  a  coefficient  of  reliability  of  .89 

for  the  rating  of  college  instructors  by  their  students.  In 

this  study  the  reliability  of  student  judgment  was  found  to  be 

greater  than  tliat  of  faculty  members.    By  the  Brown-Spearman^^ 

formula  it  was  calculated  that  it  required  the  judgments  of 

nineteen  fellow  faculty  meinbers  to  equal  the  reliability  of 

that  of  sixteen  students  concerning  the  effectiveness  of  any 

given  college  instructor. 

In  another  phase  of  this  study  the  ratings  by  the 

—  

E.  H.  Guthrie,  Up.  cit.  p.  176. 
^^Henry  E.  Garrett,  up.  cit.  p.  269. 
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students  were  divided  into  two  halves  and  the  coefficient  of 
correlation  between  the  average  rankings  for  the  first  and  sec- 
ond halves  was  found  to  be  ♦VQ.  Using  a  graphic  rating  scale, 
with  tliree  divisions  on  a  line  with  descriptions  written  below, 
he  secured  a  significantly  higher  coefficient  of  reliability, 
than  with  the  ranking  method  described  above.    He  concludes 
** there  is  considerable  agreement  among  students  concerning  the 
ability  of  their  teachers  and  that  student  opinion  is  compara- 
tively stable  from  year  to  year,**    He  also  reports  a  substan- 
tial agreement  between  ratings  given  by  under  and  upper  class- 
men, which  is  in  accord  with  the  findings  of  this  study. 

33 

Freyd    ,  experimenting  with  a  graphic  rating  scale, 
found  the  coefficients  of  correlation  between  ratings  made  dur- 
ing first  and  second  months  to  average  .76,  the  lowest  being 
.52;  and  the  coefficients  of  correlation  between  the  ratings 
made  in  the  second  and  third  months  to  average  .87,  with  the 
lowest  .66.    The  coefficient  of  correlation  between  ratings  on 

the  same  men  by  different  judges  was  found  to  be  .71. 

34 

Hemmers  and  Brandenburg     have  made  extensive  experi- 
ments in  the  use  of  the  ♦^Purdue  Rating  Scale  for  Teachers"  and 
the  evidence  is  that  this  scale  as  a  whole  has  a  reliability 
expressed  by  coefficients  of  correlation  ranging  between  .60 

and  .85  for  classes  containing  approximately  30  students.  The 

_  

qMglX  Freyd.  Op.  cit.  p.  94. 

M.  H.  Remmers  and  G.  C.  Brandenburg,  "Experimental  Data  on  the 
Purdue  Rating  Scale  for  Instructors,"  Educational  Administra- 
tion and  Supervision,  XIII  (November,  1927),  519-27. 
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reliability  of  students*  judgments  on  a  single  trait  is  report- 
ed to  be  approximately  •60  for  a  similar  number  of  judges.  By 
the  use  of  the  Spearman-Brown  formula  they  calculate  that  it 
requires  from  100  to  120  student  judgments  to  make  the  reliabil- 
ity of  their  scale  comparable  to  that  of  the  most  reliable  stan- 
dardized mental  and  educational  tests  now  available,  i.  e., 

35 

about  ,95.    Root    ,  working  with  a  checking  list  of  42  ques- 
tions relating  to  the  quality  of  instruction,  found  an  average 
coefficient  of  correlation  of  .95  between  twD  ratings  four  weeks 
apart.    The  coefficients  ranged  from  .86  to  1.0.    In  the  same 
study  he  reports  an  attempt  to  determine  the  validity  of  the 
students*   judgments  of  teaching  by  checking  the  judgments  made 
in  class  with  judgrflents  privately  expressed.    Five  students 
took  the  checking  lists  to  the  rooms  of  the  students  who  had 
previously  rated  the  instructors  in  class,  and  requested  them 
to  give  their  private  opinion,  i.  e.,  what  they  really  thought 
of  the  teachers  in  question.    The  checking  lists  were  marked  by 
the  same  means  of  identification  as  in  previous  checking  in  the 
classroom,  thus  permitting  a  comparison  to  be  made.    A  reliabil- 
ity coefficient  of  .95,  with  range  from  ,84  to  1.0,  was  found 
to  exist  between  the  two  checkings.    The  present  writer  ques- 
tions this  method  of  determining  the  validity  of  students*  judg- 
ments, since  it  was  known  to  the  raters  that  the  five  students 

who  acted  as  monitors  were  cooperating  with  the  investigator 

TrK  — 

Alfred  K.  Root,  »»Student  Ka tings  of  Teachers,**  Journal  of 
Higher  Education,  II  (June,  1931),  311-15. 
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and  that  the  results  of  the  second  rating  were  to  be  used  for 

36 

some  purpose.    Root      also  reports  very  slight  cxDrrelation  be- 
tween the  grades  given  by  the  instructor  to  the  student  and 
the  rating  given  by  the  same  student  to  the  instructor.  This 

is  in  harmony  with  the  present  study. 

37 

Fritz     ,  working  with  small  groups  totalling  89  stu- 
dents, compared  their  ratings  of  the  same  instructor  made  at 
two  periods  two  weeks  apart,     in  this  study  he  used  a  rating 
scale  for  teachers  developed  by  Dr.  V.  L.  Stricklan  d, of  Kansas 
State  Agricultural  College,  and  computed  the  degree  of  reliabil- 
ity in  terms  of  the  percentage  of  duplication  in  the  first  and 
second  ratings.    While  the  percentages  were  given  for  all  the 
separate  traits  on  the  scale,  the  average  is  sufficient  ibr  our 
purpose. 

Number  of  Estimates    Successful       Number  Number 

Duplication      varying  varying 

one  step         two  steps 

826  525  274  27 

63  per  cent    33  per  cent      3  per  cent 

38 

According  to  Fritz      ♦♦students  vary  quite  markedly  in 
their  ability  to  duplicate  a  judgment  which  they  have  once  made. 
The  per  cent  of  duplication  for  ten  individuals  chosen  at  ran- 
dom range  from  46  to  74."    He  reports  a  high  reliability  and 
concludes  that  "student  ratings  are  of  value  when  the  combined 

l^lbid.  p.  314. 

Martin  K.  Fritz,  "The  Variability  of  Judgment  in  the  Rating 
of  Teachers  by  Students,"  Educational  Administration  and 
Supervision,  XII  (December,  1926),  630-34. 
^^Ibid.  p.  634. 
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estimate  of  a  group  is  taken.    In  this  way  quite  reliable  av- 
erages for  each  group  can  he  secured.     The  average  class  is 

probably  large  enough  to  secure  a  stable  rating." 

29 

Stalmaker  and  Kemmers      found  the  following  coeffi- 
cient of  reliability  on  each  of  ten  traits  between  judgments 
of  94  students:-  .733,   ,778,   ,798,   ,824,   .852,   ,869,  .873, 
.879,    .915,  .968. 

40 

Brandenburg      reports  a  study  in  which  he  calculated 

the  reliability  coefficients  or  the  ratings  on  nine  personal 

qualities.     Each  student  was  rated  by  the  remaining  members  of 

a  group  of  25  students.     The  correlation  was  run  between  the 

combined  rating  of  one  half  the  group  with  the  combined  ratings 

of  the  otiier  half.     The  coefficients  obtained  ranged  from  .58 

to  .87  in  a  fairly  normal  distribution. 

41 

Furfey      in  experimenting  wi th  a  rating  scale  in 

which  each  trait  was  analyzed  into  several  sub-traits  viiich 

were  rated  separately  secured  a  reliability  coef f ici  ent  of 

,888  between  ratings  of  two  different  judges, 

42 

G.  B.  Watson    ,  in  what  appears  to  be  a  most  excel- 
lent summary  of  the  findings  of  experimental  work  in  the  field 
of  rating,  holds  that  rating  scales  seem  to  have  reliabilities 

J.  M.  Stalmiaker  and  Jl.  H.  Remmers,  "Can  Students  Disciiminate 
Traits  Associated  with  Success  in  Teaching"*"  Journal  of  Ap- 
plied  Psychology,  XII  (December,  1928),  602-1^]; 
G.  C.  Brandenburg,  "Analyzing  Personality,"  Journal  of  Ap- 

plied  Psychology,  IX  (1925),  146. 
P.  H.  Furfey,  "Improved  Rating  Scale  Technique,"  Journal  of 
.    Educational  Psychology.  XVII  (January,  1926),  47. 
*^G.  B.  Watson,  Op.  cit,  p.  76. 
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surpassing  raost  existing  group  tests  and  lists  the  following 

examples  in  substantiation  of  this  conclusion. 

Name  of  investigator  Reliability  coefficients 

Barr  .40-. 80 

Ji'reyd  .52 -.87 

Webb  .65-. 81 

Knight  and  Cleeton                               .80-. 90 

Shen  .62-.  91 

FvLTfey  .70-. 97 

All  of  these  studies  are  not  reported  in  detail  here 

as  they  are  not  all  in  the  field  of  student  judgment  of  college 

teachers.    But  it  is  interesting  to  note  how  closely  they  agree 

with  the  findings  made  in  the  studies  we  have  reviewed  in  more 

detail. 

In  addition  to  the  reliability  and  validity  of  a  rat- 
ing scale,  or  as  a  distinct  factor  influencing  them,  there  is 
the  phenomenon  commonly  known  as  the  "halo  effect."    This  re- 
fers to  the  tendency  for  raters  to  be  influenced  in  their  eval- 
uation of  specific  items  on  the  scale  by  their  general  attitude 
or  total  reaction  toward  the  person  being  rated.     The  extent  of 
this  spread  of  general  estimate  is  most  commonly  measured  by 
the  size  of  the  coefficients  of  correlation  between  the  differ- 
ent pairs  of  the  items  on  the  scale.    It  is  generally  held  by 
investigators  that  positive  or  negative  traits  or  qualities  are 
likely  to  be  correlated  with  one  another  and  that  therefore  we 
should  expect  to  find  positive  correlation  between  the  desir- 
able characteristics  listed  on  any  rating  scale,  but  there  is 
no  evidence  as  to  how  large  this  correlation  ought  to  be  in 
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order  to  be  indicative  of  the  presence  of  the  "halo  effect". 

Some  writers  hold  that  the  "halo  effect"  is  an  invariable  and 

unavoidable  weakness  in  any  rating  scheme  of  measuring  human 

43 

qualities.    Marsh  and  Perrin      report  practically  no  indica- 
tion of  the  existence  of  a  general  spread  of  estimate,  and 

44 

Hemmers  and  Brandenburg      arrived  at  the  same  conclusion  in 

their  extensive  experiments  with  the  Purdue  Hating  Scale  for 

45  46  47 

Teachers.     On  the  other  hand  Symonds     ,  Webb     ,  Thorndike  , 

48  49 
Rugg    ,  and  Knight  and  Franzen      offer  what  they  regard  as  con- 
clusive evidence  of  the  existence  of  the  general  spread  of  es- 
timate in  the  size  of  the  coefficients  of  correlation  which 
they  obtain.    The  conflicting  opinions  on  conclusions  of  these 
two  groups  seem  to  be  caused  by  the  lack  of  any  definite  stan- 
dard as  to  the  amount  of  intercorrelation  which  would  present 
incontestable  proof  of  the  existence  of  this  tendency,  since 
the  coefficients  of  correlation  which  the  two  groups  obtain  and 
upon  which  they  base  their  conclusions,  do  not  differ  in  size 
very  markedly.    It  is  probable  that  the  average  rater,  being 
human,  will  not  rid  himself  entirely  of  this  tendency  or  bias 
in  question,  and  exercise  to  the  full  his  analytical  powers,  un- 
less the  scale  aids  him  to  do  so.     Consequently  the  absence  of 

 — 

^TSarah  E.  Marsh  and  J?\  A.  C.  Perrin.  Op.  cit.  p.  399. 
H.  Remmers  and  G.  C.  Brandenburg.  Op.  cit  .  p.  523. 


.^P.  M.  Symonds.  Op.  cit.  p.  192. 
I'Se.  Vtfebb,  Op.  cit.  p.  55. 


L.  Thorndike,  Op.  cit.  p.  27,  29. 

0.  Rugg,  "Is  the  Rating  of  Muman  Character  Practicable?" 
Journal  of  Educational  Psychology,  XIII  (January,  1922), 
30-42,  81-93. 
^^F.  B.  Knight  and  R.  H.  Franzen,  Op.  cit.  p.  212. 
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this  «halo  effect*,  or  its  existence  in  small  amoimt  may  be 
considered  a  point  in  the  favor  of  any  given  rating  scale.  In 
other  words  it  is  possible  that  the  amount  of  "halo  effect**  is 
a  function  of  a  particular  scale  rather  than  a  necessary  fea- 
ture of  all  rating  scales.    The  writer  has  failed  to  find  any 
experimental  evidence  on  this  question,  as  to  the  size  of  cor- 
relation coefficients  which  would  establish  the  existence  or 

extent  of  '•halo  effect 

50 

Symonds     has  suggested  the  following  reasons  op 
causes  for  this  tendency  to  skew  ratings  on  every  specific 
trait  in  the  direction  of  the  total  reaction  of  the  rater  to 
the  person  being  rated,  in  so  far  as  this  tendency  may  be  ef- 
fected by  the  design  of  the  scale.    They  have  been  included 
here,  not  because  they  have  been  scientifically  established, 
but  because  in  the  absence  of  anything  more  scientific,  they 
may  serve  as  a  guide  in  the  construction  of  a  scale. 

**1.  The  trait  is  one  which  is  not  easily  observed. 

2.  The  trait  is  one  which  is  not  commonly  observed 
or  thought  about. 

3.  The  trait  is  one  which  is  not  clearly  defined. 
4*  The  trait  is  one  which  involves  reactions  with 

other  persons  rather  than  mere  personal  behavior. 
5.  The  trait  is  one  with  high  moral  importance  in 
its  usual  oonnotations*** 

In  another  study,  Symonds'''*'  has  attempted  to  deter- 
mine the  extent  to  which  the  reliability  of  a  scale  is  affected 

M.  Symonds,  Op.  cit.  p.  194,  195. 
^'*'P.  M.  Symonds,  •»0n  the  Loss  of  Reliability  in  Ratings  Due  to 
Coarseness  of  the  Scale,*  Journal  of  Experimental  Psychology, 
YII  (December,  1924),  459* 
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by  the  •coarseness  of  grouping,**  i,  e«,  the  number  of  divisions 
or  gradations  on  the  scale.    He  briefly  reviews  studies,  made 
by  various  noted  experimenters,  among  them  Gal ton  and  Pearson, 
in  which  the  number  of  divisions  on  the  scale  varied  from  five 
to  eleven.    The  average  coefficient  of  reliability  obtained  by 
all  experimenters  was  ♦SS*    Symonds  believes  that  a  coefficient 
of  reliability  of  ,60  is  about  as  hi^  as  can  be  expected  in 
rating  human  qualities  and  reasons  that  this  fixes  the  optimum 
number  of  divisions,  according  to  his  statistical  calculations, 
at  seven* 


CHAPTER  II 

DEFINITIOII  AND  LBIITATIOKS  OF  THE  PROBLEM 


The  investigation  reported  in  this  dissertation  has  to 
do  with  the  improveraent  of  instruction  in  institutions  of  high- 
er learning.     The  central  problem  is  the  construction  and  test- 
ing of  a  reliable  and  valid  instrument  for  the  measureraent  and 
diagnosis  of  the  teaching  ability  and  personal  fitness  of  col- 
lege and  university  instructors. 

In  addition  to  reliability  and  validity  the  instrument 
sought  must  satisfy  another  requirement.     In  order  to  be  of 
practical  value  in  the  institution  for  which  it  was  originally 
designed,  Iowa  State  College,  and  in  other  colleges  and  uni- 
versities of  like  size,  its  adrainistration  must  be  sufficient- 
ly economical  of  both  time  and  expense  to  permit  of  its  use  on 
a  large  scale. 

Not  a  few  minor  problens  relevant  to  the  extended  use 
of  the  instruiTient  v/hich  was  finally  developed,  i.e..  The  Col- 
lege Teachers  Rating  Scale,  were  encountered  during  the  in- 
vestigation.    Four  of  these  were  considered  to  be  of  sufficient 
importance  to  warrant  efforts  being  expended  upon  their  solu- 
tions, and  were  included  in  this  study.     These  four  minor  prob- 
lems selected  for  study  are  formulated  in  the  following  state- 
ments : 

1.  Viliat  relation  exists  between  the  ability  of  the  stu- 
dent and  the  rating  which  he  gives  his  instructor? 
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2.  What  relation  exists  between  the  maturity  of  the 
student  and  the  rating  which  he  gives  his  instructor? 

3.  What  relation  exists  between  the  size  of  the  class 
and  the  average  rating  received  by  its  instructor? 

4.  V/hat  is  the  trend  in  the  numerical  values  of  the 
scores  received  by  individual  instructors  in  success- 
ive ratings? 

Many  schemes  and  devices  for  evaluating  the  teacher 
and  his  work  have  been  designed  and  applied,  particularly  in 
the  elementary  and  secondary  schools,  where  in  most  cases 
trained  supervisors  or  administrators  direct  the  application 
of  the  particular  form  of  measurement  used.l    These  instru- 
ments for  measuring  teaching  efficiency  fall  into  two  general 
classes,  i.e.,   (1)  objective  tests,  usually  of  the  standardized 
variety,  and  (2)  those  measures  of  more  subjective  nature,  in- 
cluding check  lists,  questionnaires  and  rating  scales.    The  ob- 
jective tests  are  administered  at  the  beginning  and  at  the  end 
of  the  teaching  period,  and  the  difference  between  the  first 
and  last  scores  is  taken  as  the  result  of  the  instruction  of 
the  teacher  in  question.^ 

This  method,  however,  is  not  accepted  by  many  educators 


'A.  S.  Barr  and  V/illiam  H.  Burton,  The  Supervision  of  Instruc- 

,    tion.    New  York:  D.  Appleton  and  Co.,  1926.   pp. 468-86, 

'E.  S.  Jones,   "Suggestions  for  Teacher  Measurement",  Schoox  and 

Society,  VI  (September  15,  1917),  321. 
N.  V.  Scheidemann,  "Estimating  the  College  Teacher's  Teaching 

Ability",  School  and  Society,  iXVI,   (December  3,  1927), 

pp.  717-718. 
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on  the  ground  that  objective  tests  fail  to  measure  all  the  de- 
sirable and  valuable  outcomes  of  teaching. 

The  instruiaents  of  the  more  subjective  type  vary  con- 
siderably in  form  and  in  the  manner  in  which  they  are  adminis- 
tered.   The  various  forms  in  use  may  be  roughly  classified  as 
check  lists,  questionnaires  and  rating  scales.^    Certain  of 
these  forms  are  designed  to  be  used  by  a  supervisor  or  adminis- 
trator, who,  after  observing  the  teacher  at  work,  records  upon 
the  form  being  used,  his  estimate  of  the  teaching  ability  and 
personal  fitness  of  the  instructor  in  terms  of  the  particular 
items  inscribed  upon  it.^    Because  this  form  of  measurement 
absorbs  the  time  of  a  high-salaried  officer,  considerable  ex- 
pense is  necessarily  entailed  in  its  use.     Other  devices  are 
designed  to  be  used  by  teachers  in  self  evaluation  and  analy- 
sis,^ while  still  others  are  based  upon  student  judgment  or 
opinion.     These  last,  while  not  so  numerous  as  the  other  types 
mentioned,  have  been  used  in  many  institutions.     In  a  recent 
summary  of  the  current  efforts  being  made  to  improve  instruc- 
tion in  colleges  the  reader  is  informed  that  "the  plan  of  ask- 

•^erschel  T.  Manuel,  The  Use  of  Quantitative  Measurement  by 
Members  of  the  National  Society  of  College  Teachers  of 
Education  and  Their  Attitude  Toward  It.     Yearbook  Number 
XVIII  of  the  National  Society  of  College  Teachers  of  Educa- 
tion, 1930.     Ghicagp:  University  of  Chicago  Press,  1930. 
Chapter  VII,  pp.  178-89. 
^A.  S.  Barr  and  William  H.  Burton,  Op.  cit.,  pp.  458-86, 
^A.  S.  Barr  and  William  H.  Burton,  Op.  cit.,  p.  467, 
^Bertha  Y.  Hebb,  "Samples  of  Teachers  Self -Rating  Scales." 

City  School  Leaflet  No,  18.  Washington,  D.  C.  U.  S.  Bureau 
of  Education.     February,  1925,  1-15, 
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ing  students  to  appraise  the  v/ork  of  instructors  was  reported 
by  many  teacher-training  institutions.    At  times,  students 
prepare  reports  without  the  aid  of  any  detailed  directions;  at 
other  times,  they  are  provided  with  sets  of  questions  relating 
to  instructors  and  the  effectiveness  of  their  teaching  activi- 
ties; not  infrequently  the  use  by  students  of  the  Purdue  Rating 
Scale  was  reported."'''    a  summary  of  the  use  of  student  judg- 
ment concerning  problems  of  higher  education  is  reported  by 
Patton,^  who  believes  that  "there  is  a  place  and  a  need  for  the 
discreet  use  of  reports  by  students,  of  enlisting  their  cooper- 
ation in  surveys  and  investigations  which  seek  to  improve  the 
philosophy  and  technique  of  higher  education." 

Various  arguments  against  the  rating  of  college  teach- 
ers by  their  students  have  been  advanced,  but  an  investigation 
of  the  literature  leads  one  to  believe  that  those  who  are  most 
bitter  in  their  denunciation  of  the  plan  have  not  experimented 
with  it,  or,  if  they  have,  have  failed  to  publish  reports  of 
their  investigations.    The  relatively  few  workers  who  have  stud- 
ied student-rating  in  a  scientific  manner  report  more  favorably 
upon  its  use,  as  the  review  of  literature  in  the  preceding  chap- 
ter discloses. 

"^Stuart  A.  Courtis,  Current  Efforts  to  Improve  Instruction  in 
Schools,  Colleges  and  Departments  of  Education  and  in  Teach- 
ers Colleges.    Yearbook  Number  XVII  of  the  National  Society 
of  College  Teachers  of  Education,  1929.     Chicago:  University 

Q    of  Chicago  Press,  1929.     p.  43. 
Leslie  K.  Patton,   "Undergraduate  Student  Reports,"  Journal  of 
Higher  Education,  III  (June,  1932),  pp.  285-293. 
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There  are  those  who  advocate  the  rating  of  college 
teachers  by  their  students,  not  on  the  ground  that  such  rating 
is  a  direct  measure  of  the  quality  of  the  instruction,  but  be- 
cause it  is  a  measure  or  an  indication  of  the  student's  atti- 
tude toward  his  instructor.     It  is  held,  by  those  who  take  this 
view,  that  the  attitude  of  the  student  toward  his  instructor  is 
a  factor  of  very  great  importance  in  the  effectiveness  of  the 
teaching  process.^    There  would  appear  to  be  a  well-grounded 
opinion  on  the  part  of  those  persons  who  have  studied  this 
question  scientifically  that  "in  any  campaign  for  the  improve- 
ment of  college  instruction  the  students  of  the  college  have 
something  important  to  contribute  and  should  be  permitted  to 
contribute  it."l^ 

However,  it  is  not  even  one  of  the  minor  purposes  of 
this  dissertation  to  discuss  the  "pros"  and  "cons"  of  the 
practice  of  having  college  teachers  rated  or  judged  by  their 
students,  even  though  certain  rather  definite  convictions  con- 
cerning the  same  may  have  resulted  from  the  experience  v/ith 
the  particular  scale  described  herein.     The  foregoing  state- 
ments have  been  submitted  as  justification  for  the  energy  and 
time  expended  upon  this  attempt  to  study  the  problem  in  a 
scientific  manner.     It  is  the  aim  of  the  writer  to  present 

G.  Brandenburg  and  H.  H.  Remmers,  The  Use  of  Teacher 
Rating  at  Purdue  University.    Purdue  University  School  of 
Education  Bulletin,  Vol.  VI,  No.  6:  77.     LaFayette:  Purdue 
^      "University.  1928. 

-•■^F.  S.  Breed,  "A  Guide  for  College  Teaching,"  School  and 
Society,  XXIV  (July  17,  1926),  p.  84. 
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statistical  evidence  of  the  reliability  and  validity  of  the 
one  scale,  v/hich  was  designed  and  tested  in  this  investigation, 
along  with  a  careful  and  accurate  description  of  the  procedure 
followed  in  the  determination  of  the  same, 

iVfter  a  careful  study  of  the  various  schemes,  in  use 
for  measuring  the  effectiveness  of  teaching,  and  v/ith  their 
peculiar  advantages  and  disadvantages  in  mind,  it  was  decided 
to  attempt  the  construction  of  a  scale  upon  which  the  student 
would  indicate  his  evaluation  of  the  teaching  ability  and  per- 
sonal fitness  of  his  instructor.     The  construction  and  testing 
of  such  a  scale  was  selected  as  the  central  problem  of  the  in- 
vestigation reported  in  this  dissertation. 

Having  conceived  the  problem,  it  became  evident  that  a 
rather  extensive  investigation  would  be  required  for  its  solu- 
tion.    The  opportunity  to  conduct  such  an  investigation  came 
at  Iowa  State  College  through  the  cooperation  of  the  president 
of  the  institution  and  his  newly-created  Council  on  the  Improve 
ment  of  Teaching.     This  council,  appointed  in  September,  1928, 
was  commissioned  to  initiate  an  active  and  wide-spread  campaign 
to  improve  the  quality  of  the  instruction  within  the  institu- 
tion.    The  writer  secured  an  invitation  to  cooperate  with  the 
council  and  to  undertake  the  design  and  testing  of  a  scale  for 
measuring  the  quality  of  teaching  as  his  special  problem.  This 
arrangement  offered  to  the  investigator  a  faculty  of  over  400 
members,  a  student  body  of  approximately  4,000  full-time  stu- 
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dents,  and  an  active  interest  developed  through  several  years 
in  the  improvement  of  college  teaching,  within  its  own  walls. 
The  fact  that  the  writer  enjoyed  full  membership  on  the  teach- 
ing staff  of  the  college,  the  cooperation  of  the  president, 
and  the  Council  on  Teaching,  made  it  most  convenient  to  carry 
on  an  extensive  and  comprehensive  objective  investigation. 

These  advantages  however,  while  real  and  important, 
carried  with  them  the  necessity  of  meeting  the  ideas  of  the 
president  of  the  college  and  the  Council  on  Teaching.  This 
influenced  slightly  the  selection  of  the  traits  included  in 
the  scale  ajid  at  times  compelled  the  investigator  to  do  con- 
siderable work  which  was  not  closely  related  to  tne  central 
problem  of  this  study. 


CHAPTER  III 
THE  GENERAL  STORY  OF  THE  INVESTIGATION 

The  general  character  of  the  proposed  scale  having 
been  decided  upon  (See  Chapter  II),  the  actual  construction  of 
it  becarie  the  next  step.     This  problem  resolved  itself  into 
two  main  questions: 

1.  V/hat  items  in  the  way  of  professional  and  personal 
qualities  should  be  listed  on  the  proposed  scale? 

2.  What  specific  type  or  form  of  scale  should  be 
adopted? 

A  review  of  the  literature  of  rating  (See  Chapter  I), 
indicates  that  neither  of  these  questions  have  been  conclusive- 
ly answered  by  strictly  objective  methods  of  investigation 
being  still  quite  largely  matters  of  subjective  judgment,! 
This  is  manifested  by  the  variations  which  exist  in  the  lists 
of  qualities  discovered  by  the  different  investigators,  whose 
findings  are  reported  in  Chapter  I,  and  by  the  conflicting  con- 
clusions concerning  the  virtues  of  the  different  types  of 
scales.     It  should  be  noted,  however,  that  there  are  certain 
items  which  appeared  in  a  large  proportion  of  the  lists  sub- 
mitted by  the  various  investigators.    Whatever  authority  may 
be  attributed  to  a  consensus  of  opinion  might  be  employed  as 

■^National  Education  Association,  Division  of  Research, 

"Practices  Affecting  Teacher  Personnel."    Research  Bulletin 
6:239;  September,  1928.     Washington,  D.  C;  the  Association. 
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evidence  of  the  importance  of  those  items  which  appear  in 
several  different  lists. 

Aided  by  the  studies  reported  in  Chapter  I,  and  with 
the  limitations  of  the  situation  well  in  mind,  this  investiga- 
tor, with  the  cooperation  of  the  Council  on  Teaching,  selected 
the  following  items. 

1.  Preparation  for  class  meetings 

2.  Interest  and  enthusiasm  in  his  subject 

3.  Ability  to  arouse  interest  in  subject 

4.  Feeling  between  instructor  and  students 

5.  Organization  of  course 

6.  Thinking  demanded  of  students 

7.  Ability  to  express  thought 

8.  Enunciation 

9.  Leading  discussion  and  questioning 

10.  Personal  appearance 

11.  Personal  peculiarities 

12.  Interest  in  students 

13.  Sense  of  proportion 

14.  Sense  of  humor 

15.  Self-confidence 

16.  Tolerance  and  liberality 

This  investigator  is  unable  to  submit  any  conclusive 
evidence  that  the  items  in  the  foregoing  list  represent  the 
most  essential  abilities  and  qualities  of  the  successful  col- 
lege instructor.     It  may  be  observed,  however,  that  they  ap- 
pear frequently  in  the  lists  reviewed  in  Chapter  I,  and  that 
practically  all  are  to  be  found  under  one  name  or  another  in 
the  list  reported  by  F.  S.  Breed  (See  page  2   ).     The  careful 
method  employed  in  determining  the  items  in  Breed's  list,  has 
commended  it  to  other  investigators. 2 


H.  Rammers,   "The  College  Professor  as  the  Student  Sees 
Him."    Studies  in  Higher  Education,  XI,  Purdue  University 
Bulletin,  XXIX,  No.  6.  p.  17.     LaFayette , Indiana:  Purdue 
University.  1929. 
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The  next  decision  had  to  do  v/ith  the  type  or  form  of 
scale  to  employ  in  this  investigation.     A  careful  study  of  the 
various  types,  proposed  and  in  actual  use,  was  made  and  after 
repeated  attempts  to  adapt  each  of  them  to  the  needs  of  the 
situation,  it  was  decided  to  adopt  the  "graphic"  form.  In 
this  type  of  scale  a  line  is  used  to  indicate  the  full  range 
of  each  characteristic  to  be  measured.     Appropriate  statements 
describing,  in  terms  as  objective  as  possible,  the  varying 
gradations  of  each  characteristic,  are  written  under  each  line. 
The  rater  is  requested  to  place  a  check  mark  at  that  point  on 
each  line  v/hich  represents  the  extent  to  which  the  person 
being  rated  possesses  the  characteristic  represented  by  the 
line. 

Great  care  was  exercised  in  the  formulation  of  these 
descriptive  statements  as  it  was  fully  appreciated  that  the 
reliability  and  validity  of  the  completed  scale  would  depend 
largely  upon  their  clarity  and  objectivity. 

The  scale  which  grew  out  of  these  first  attempts  was 
experimented  with  in  the  writer *s  o?/n  classes.     This  resulted 
in  a  few  minor  changes  being  made,  to  produce  the  form  used  in 
the  first  general  rating,  which  was  made  at  the  end  of  the  fall 

^G.  C.  Brandenburg  and  E.  H.  Remmers,   "Rating  Scales  for 

Instructors,"    Educational  Administration  and  .Supervision, 
XIII  (September,  1927),  399-406. 
D.  G.  Patterson,  "Methods  of  Rating  Human  Qualities,"  Annals 
of  the  American  Academy  of  Political  and  Social  Science, 
CX  (1923),  81-93. 
Bruce  Moore,  "Personnel  Selection  of  Graduate  Engineers," 
Psychological  Monograph,  XXX  (1921),  84-90. 
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term  of  1928.     A  copy  of  this  scale  is  filed  in  Appendix  A 
(See  Exhibit    1     ).     For  this  rating  the  writer  secured  the 
cooperation  of  fifty  instructors,  who  agreed  to  administer  the 
scale  in  their  own  classes  according  to  instructions.  They 
were  instructed  to  distribute  the  scales  and  envelopes,  one  of 
each  to  each  student,  without  comment,  other  than  a  request 
that  the  students  cooperate  and  follow  closely  the  instructions 
on  the  scale.     The  instructor  then  left  the  room,  having  ap- 
pointed a  member  of  the  class  to  collect  the  sealed  envelopes, 
and  to  carry  the  same  to  the  office  of  the  writer. 

The  scales  were  examined,  graded,  and  the  averages  of 
the  ratings  on  each  item,  along  with  the  range  of  the  ratings 
received  on  each  trait,  were  reported  to  the  instructor  con- 
cerned.    The  Hollereth  tabulating  machine  and  skilled  operators 
of  the  same  were  used  for  this  work  of  computing  averages,  and 
every  attempt  was  made  to  have  the  calculations  accurately  made 
and  reported. 

Careful  study  of  the  findings  of  this  first  survey,  the 
details  of  which  are  reported  in  Chapter  V,  disclosed  certain 
weaknesses  in  the  wording  and  arrangement  of  the  scale,  and 
suggested  certain  minor  changes.     These  changes  were  incorpo- 
rated into  the  revised  edition,  a  copy  of  which  is  shown  on 
page  39 . 

Twenty  thousand  copies  of  this  edition  of  the  scale 
were  printed  and  plans  were  laid  to  carry  on  a  college-wide  sur 
vey  during  the  last  two  weeks  of  the  winter  term,  1929. 


RATING  SCALE  FOR  TEACHERS 


-Course  No. 


Print  Instructor's  name  here  ,  Dep't  

Please  rate  your  instructor  as  to  each  of  the  seventeen  qualities  named  below. 

It  is  necessary  in  each  case  only  to  place  a  check  mark  ( V )  on  the  accompanying  line  at  that  point  which  seems  to  you  to  be 
right. 

The  higher  ratings  are  made  by  placing  the  check  mark  at  the  left  end  of  the  line;  the  lower,  by  placing  it  at  the  right.  These 
stages  and  also  the  middle  ones,  are  clearly  described  by  the  words  printed  just  below  the  line.  The  check  mark  may  in  any  given 
case  be  placed  either  directly  above  the  words  or  along  the  line  between  them  as  you  deem  proper. 

Please  make  this  rating  conscientiously  and  individually.  Your  instructor  will  receive  only  the  final  summary  of  the  results  from 
the  class  as  a  whole.   He  will  use  them  for  self-improvement  in  teaching. 


I.    Preparation  for  class 
meetings 

II.   Interest  and  enthusiasm  in 
his  subject 

ni.   Ability  to  arouse  interest  in 
students 

rV.  Organization  of  course 

V.  Scholarship 

VI.  Ability  to  express  thought 

VII.  Enunciation 

VIII.    Thinking  demanded  of 
students 

IX.  Assignments 

X.   Leading  discussion  and 
questioning 

XI.   Sense  of  proportion 

Xri.    Feeling  between  instructor 
and  students 

XIII.  Sense  of  humor 

XIV.  Self-confidence 

XV,    Tolerance  and  liberality 

XVI.  Personal  appearance 

XVII.  Personal  peculiarities 


Class  meetings  very  carefully 
planned. 

Usually  some  preparation ;  often 
seems  inadequate. 

Little  or  no  preparation. 

Very  enthusiastic  and 
interested. 

Seems  only  mildly  interested. 

Subject  seems  irksome  to  him. 

Interest  usually  runs  high. 

Students  seem  only  mildly 
interested. 

Majority  inattentive  most  of 
period. 

Course    well-organized ;  parts 
clearly  related. 

Some  organization,  but  not 
always  clear. 

Little  or  no  organization. 

Knowledge  of  subject  broad 
and  accurate. 

Knowledge  apparently  deficient 
at  times. 

Knowledge  very  plainly  defi- 
cient. 

Words  come  easily ;  meaning 
always  clear. 

Some    hesitation    for    words ; 
meaning  at  times  not  clear. 

Much  hesitation  for  words, 
meaning  often  not  clear. 

Speaks  very  clearly  and 
distinctly. 

Words  sometimes  indistinct  and 
not  easy  to  hear. 

Words    very    ilndistinct ;  oft43n 
impossible  to  hear. 

Work  demands  much  sound, 
original  thinking. 

Thinking  and  memorization 
required  about  equally. 

Thinking    discouraged ;  much 
memorization  demanded. 

Assignments    clear,  reasonable 
and  carefully  given. 

Katiier  indefinite  and  often 
hurriedly  given. 

Very  indefinite,  usually  hurried- 
ly given. 

Questions  thought-provoking, 
discussions  lively  and  worth- 
while. 

Questions  usually  call  for  facts 
or  lead  to  rambling  discussion. 

Few  questions  or  none  put  to 
class. 

Stresses    important    topics ; 

Occasionally    stresses  details, 
neglecting  important  topics. 

Often  neglects  important  topics 

Feeling  of  good-will  prevails 
strongly. 

Neither  good-will  nor  antagon- 
ism seems  to  prevail. 

Instructor  tends  to  antagonize 
class. 

Has  keen  sense  of  humor. 

Humor  occasionally,  but  not 
often  exhibited. 

Manifests  little  or  no  humor. 

Sure  of  himself ;  meets  ditlicul- 
ties  with  poise. 

Fairly   self-confident ;  occasion- 
ally disconcerted. 

Hesitant,    timid,  uncertain. 

Welcomes  dirteiences  oi  opinion. 

Sometimes  impatient  when  stu- 
dents oppose  his  views. 

Easily  aroused  to  temper  by 
opposition. 

Well-groomed ;  clothes  neat, 
clean,  in  good  taste. 

Usually  rather  untidy  and  care- 
less as  to  appearance. 

Slovenly ;  clothes  and  person 
untidy. 

Manner  pleasing ;  free  from  an- 
noying mannerisms. 

Objectionable  mannerisms  not 
serious  or  numerous. 

Constantly  exhibits  annoying 
mannerisms. 

Please  do  not  sign  your  name  or  make  any  other  mark  which  might  serve  to  identify  you.  Place  this  sheet  in  envelope,  seal  and  give  It  to  instructor.  He 
will  forward  it  to  the  council  on  teachins  without  opening  it.  27764 
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Similar  surveys  were  made  at  the  end  of  each  succeed- 
ing term,  with  but  slight  variation  in  the  general  procedure. 
The  slight  changes  which  were  made  from  term  to  term  in  ad- 
ministering the  ratings  were  chiefly  refinements  in  the  tech- 
nique of  handling  the  mechanics  of  the  problem  and  were  not  of 
such  a  nature  as  would  affect  the  numerical  results. 

The  first  experience  with  the  scale  in  the  fall  of 
1988,  had  disclosed  the  necessity  of  preparing  very  definite 
instructions  to  guide  instructors  in  the  administration  of  the 
scale  in  their  own  classes.     Verbal  instructions  of  a  general 
nature  had  been  given  to  those  instructors  who  participated  in 
the  first  rating,  but  it  was  discovered  that  several  of  them 
had  disobeyed  instructions  in  what  was  evidently  an  attempt  to 
secure  a  favorable  rating  from  their  students.    Written  instruc- 
tions of  a  very  specific  nature  were  prepared  and  used  in  sub- 
sequent surveys.     These  instructions  included  a  carefully  word- 
ed statement  for  the  instructor  to  read  aloud  to  his  class  be- 
fore distributing  the  scales.    Every  attempt  was  made  to  make 
the  conditions  of  rating  in  the  different  classes  as  nearly 
uniform  as  possible.    A  copy  of  these  instructions  is  filed  in 
the  Appendix  (See  Appendix  A,  Exhibit  3). 

When  the  findings  of  each  survey  had  been  computed,  a 
report  was  sent  to  each  instructor  for  each  of  the  classes 
rating  him.    After  considerable  experimentation  in  recording 
and  reporting  results,  a  standard  form  was  adopted,  v/hich  gave 
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the  average  and  the  distribution  of  the  scores  on  each  sepa- 
rate trait.     A  specimen  of  such  a  report  is  submitted  on 
page  42,  This  is  a  copy  of  an  actual  report,  v/ith  only  the 
name  of  the  instructor  and  of  the  course  changed. 

In  order  to  assist  the  instructor  in  the  interpreta- 
tion and  use  of  the  reports  of  their  ratings,  a  summary  of  the 
significant  findings  of  each  of  the  first  three  surveys  was 
made  at  the  completion  of  each  survey  and  issued  to  all  in- 
structors participating. 

A  brief  summary  of  the  different  rating  surveys  which, 
were  carried  on  in  this  investigation,  with  the  number  of 
teachers,   classes,  and  students  involved  in  each,  is  presented 
in  the  following  table. 

TABLE  I 

NUl'ffiER  0?  INSTRUCTORS,  CLASSES  AIID  STUDENTS  INYOLVED 
IN  THE  CRITICAL  EVALUATION  OF  THE 
COLLEGE  TEACHER »S  RATING  SCALE 


Quarter 

Number 

Number 

Number 

Mean  Number 

Mean  Number 

or 

of 

of 

of 

of  Classes 

of  Students 

Term 

Instruc- 

Classes 

Students 

per 

per  Class 

tors 

Instructor 

Fall  1928 

49 

94 

1654 

1.88 

17.59 

V/inter  1929 

203 

466 

8464 

2.29 

18.20 

Spring  1929 

28 

49 

998 

1.75 

20.36 

Summer  1929 

16 

11 

126 

1.75 

11.45 

Fall  1929 

128 

245 

4695 

1.89 

19.33 

Winter  1930 

111 

209 

4173 

1.86 

20.25 

Spring  1930 

27 

40 

857 

1.48 

21.42 

Summer  1930 

8 

10 

176 

1.25 

17.60 

Fall  1930 

92 

151 

3163 

1.65 

20.94 

Winter  1931 

51 

89 

1768 

1.74 

19.86 

Totals 

703 

1364 

26,074 

1.94 

19.18 

-42- 


REPORT  OF  RATING  BY  CLASS 


Instructor      Doe,  John  R»  Course       Philosophy  45 

Quarter    Winter  Year 1930  No.  in  class  15 


1 

TRAITS  J 
1 

Distribution  of  grades  given  by  students: 

age 

90-: 
,100: 

80- : 70- J 60-; 
89  :79  i69  : 

50- J 40-: 30-: 
59:  49:  39: 

20-: 10-  : 
29:  19  : 

0-: 

9  : 

I.  Preparation  for      '  g  ' 
class  meetings        *  ' 

8  !  1  i  1  ! 

•             •  I 

•  •  ■ 

•  •  1 

•  •  a 

•  •  a 

• 
• 

86 

II,  Interest  and  enliuw 
iasm  in  his  subject: 

*i5 : 

•  •  1 

•  •  1 

•  •  a 

•  •  a 

•  •  a 

•  •  a 

•  •  a 

•  •  a 

•  « 

•  1 

•  a 

•  1 

III.  Ability  to  arouse  i 
interest  in  stu-  i 

1  i 

:  : 
6  :  1  ! 
:  : 

: 

4  i 

: 

3  : 

J 

: 

.  74 

IV.  Organiiation  of  ! 
course  i 

2  , 

5  ; 

4  ; 

3  ; 

!  1  \ 

■ 

;  7« 

V.  Scholarship  j 

9  ; 

6  i 

:  : 

: 

!  91 

VI.  Ability  to  express 
thought 

!  0 

5  I 

'  6  ; 

4  ' 

76 

VII.  Enunciation  i 

.  5 

i  8  . 

,  1  : 

0  ! 

.  1  i 

I  i 

:  36 

VIII.  Thinking  demanded  , 
of  students 

'10 : 

.  1  . 

1  ' 

!    ^  \ 

9 

■ 

87 

IX.  Assignments  ; 

,  7 

!  2  s  4  : 

1  . 

!    1  i 

: 

:  84 

X.  Leading  discussion 
and  questioning 

;  6 

;  3  ' 

'  2  ; 

'  4 

\  i 

;  82 

XI.  Sense  of  proportin 

!  5 

:  3 

\    5  ! 

,  2 

!  82 

XII.  Feeling  between 
instructor  and 
students 

:  6 

t  3  ' 

;  3 

1 

!    2  i 

\  82 

XIII.  Sense  of  humor 

sll 

i  2 

i  1 

i  1 

\  90 

XIV.  Self-confidence 

:  7  J  6 

!  2 

!  88 

XV.  Tolerance  and  lib- 
erality 

;  8 

;  7 

;  90 

XVI.  Personal  appear- 
ance 

[  3 

[  5 

'  5 

[  0 

'  2 

\  80 

XVII.  Personal  peculiar- 
ities 

;  5 

;  3 

;  4 

!  3 

;  82 

COUNCIL  ON  TEACHING. 
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The  reader  should  note  that  the  total  "703"  does  not 
represent  703  individual  teachers,   since  raany  of  them  v/ere  rated 
several  times.     This  repeated  rating  of  many  instructors  car- 
ried many  advantages,  as  it  enabled  the  investigator  to  study 
the  changes  in  the  ratings  received  by  the  same  instructor  at 
different  times  and  in  classes  of  varying  sizes.     The  spread 
in  the  sizes  of  classes  was  very  considerable,  ranging  from  2 
to  82.     This  enabled  the  investigator  to  study  the  influence  of 
class  size  upon  the  ratings.     The  findings  of  this  particular 
phase  of  the  investigation  are  presented  in  Chapter  IX* 

The  number  of  classes  per  teacher  per  quarter  is  not 
very  clearly  represented  by  the  weighted  average  (1.94)  since 
it  ranged  from  one  to  five,  v/ith  a  distribution  as  follows: 
496  teachers  were  rated  by  1  class;  584  teachers  were  rated  by 
2  classes;  167  teachers  were  rated  by  3  classes;  97  teachers 
were  rated  by  4  classes;  15  teachers  were  rated  by  5  classes. 

After  the  winter  quarter  of  1929,  teachers  v/ere  re- 
quested to  limit  the  number  of  classes  rating  them  to  two,  in 
order  to  lighten  the  work  involved  in  grading  the  scales,  and 
in  tabulating  and  reporting  the  results.     This  accounts,  in 
part,  for  the  decrease  in  the  nuiaber  of  scales  used  in  surveys 
subsequent  to  that  of  winter,  1929. 

As  the  final  work  on  the  organization  and  interpreta- 
tion of  the  data  was  begun  at  the  end  of  the  winter  quarter  of 
1931,  ratings  made  in  subsequent  quarters  are  not  reported  in 
this  dissertation,  although  proving  of  value  in  certain  phases 
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of  the  study. 

In  this  chapter,  only  the  general  outlines  of  the  in- 
vestigation are  submitted.    More  detailed  description  and 
analysis  of  the  techniques  used  in  the  various  specific  phases 
of  the  investigation  will  be  presented  in  the  chapter  immediate 
ly  following. 


CHAPTER  IV 

DESCRIPTION  AND  CRITICAL  EVALUATION 
OF  THE  TECHNIQUES  EIvIPLOYED 

This  chapter  is  devoted  to  the  description  and  criti- 
cal evaluation  of  the  techniques  employed  in  the  development 
and  testing  of  a  rating  scale  for  measuring  the  teaching  abili- 
ty and  personal  fitness  of  college  instructors,  by  the  use  of 
student  judgment  or  opinion.     This  scale  will  be  referred  to 
hereafter  in  this  dissertation  as  the  "College  Teachers  Rating 
Scale".     The  various  techniques  employed  will  be  described  and 
evaluated  in  the  order  in  which  they  were  used  in  the  investi- 
gation. 

1.  Design  and  Construction  of  the  Scale. 

The  first  step  in  the  construction  of  the  scale  in- 
volved a  careful  study  of  the  literature  of  rating,  in  general, 
and  of  the  rating  of  college  teachers  by  their  students,  in 
particular.     Only  a  small  proportion  of  the  literature,  per- 
taining to  the  latter  dealt  with  the  findings  of  scientific  ob- 
jective investigation  and  this  small  proportion  is  confined  to 
the  question  of  the  reliability  of  student  judgraent.     The  liter- 
ature records  that  isolated  attempts  have  been  made  in  various 
colleges  to  determine  the  most  essential  abilities  and  personal 
qualities  of  the  successful  college  teacher.     The  techniques 
used  in  these  studies  consist,  for  the  most  part,  in  obtaining 
the  opinion  of  college  students,  of  alumni,  and  of  college 
teachers,  as  to  the  qualities  which  they  consider  most  essential 
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in  the  successful  college  teacher.    The  scientific  character 
of  such  a  technique  might  well  be  challenged,  but  such  is  not 
the  purpose  of  this  study.     Because  of  the  pseudo-scientific 
nature  of  these  last-mentioned  investigations  and  the  lack  of 
correspondence  in  the  premises  and  techniques  employed,  the 
findings  of  the  different  studies  reported  do  not  admit  of  con- 
solidation and  close  comparison.     The  studies  referred  to 
above  have  been  quite  fully  reviewed  in  Chapter  I. 

In  the  determination  of  the  items  of  the  proposed 
scale,  this  investigator  and  the  Council  on  Teaching  were  guided 
to  some  extent  by  the  findings  of  the  studies  referred  to  above 
and  by  the  published  opinions  of  college  administrators,  sup- 
plemented by  their  own  judgment  and  experience.     It  will  be 
noted,  however,  that  the  items  finally  selected  appear  more  or 
less  frequently  in  the  different  lists  reported  in  Chapter  I. 

A  second  factor  which  conditions  the  selection  of  the 
items  for  a  rating  scale  is  the  necessity  of  limiting  their 
number.     There  seems  to  be  no  agreement  as  to  the  optimuia 
number  of  items  and  Watson^  in  his  extensive  suraraary  of  what 
has  been  discovered  about  rating  during  the  past  twenty  years 
fails  to  mention  this  question.    According  to  Remmers  "it  has 
been  found  through  considerable  experimentation  that  it  is  not 
desirable  to  use  in  such  rating  schemes  more  than  about  ten 


G.  B.  Watson,  "Supplementary  Review  of  Measures  of  Personality 
Traits,"    Journal  of  Educational  Psychology,  XVIII 
(February,  1927),  73. 
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separate  items  or  traits."*^    Fryer  is  even  more  conservative 
and  holds  that  "three,  or  five  traits  at  the  most  is  the  niim- 
ber  that  appears  to  achieve  the  greatest  accuracy  under  the  con- 
ditions of  the  rating  scale.        It  should  be  noted  that  Fryer *s 
statement  pertains  to  the  rating  of  industrial  workers  by  their 
superiors.    Both  Fryer  and  Remmers  fail  to  cite  authority  for 
their  statements  and  this  investigator  failed  to  discover  in 
his  review  of  the  literature  anything  more  definite  on  this 
question  than  a  substantial  agreement  among  students  of  rating 
that  a  large  number  of  items  operates  against  the  reliability 
and  validity  of  the  scale.    Because  of  the  absence  of  scien- 
tific data  on  the  optimum  number  of  items  in  a  rating  scale  it 
was  thought  permissible  to  use  seventeen  items  in  the  College 
Rating  Scale.     This  nuraber  is  purely  arbitrary  and  was  ex- 
tended beyond  ten  in  order  to  include  certain  items  v;hich  were 
listed  in  several  of  the  studies  reported  in  Chapter  I,  and  con- 
sidered by  this  investigator  and  by  the  Council  on  Teaching  as 
being  too  important  to  omit. 

The  following  items  were  finally  selected  and  incorpo- 
rated into  the  College  Rating  Scale  in  the  order  listed  below. 
I.  Preparation  for  class  meetings. 

H.  Remmers,  "The  College  Professor  as  the  Student  Sees 
Him,"  Studies  in  Higher  Education,  XI.    Purdue  University 
Bulletin,  Vol.  XXIX,  No.  6.     Lafayette,  Indiana;  Purdue 
University,  1929.     pp.  17-18. 
Douglas  Fryer,  "Rating  a  'Rating  Scale'"    Industrial  Manage- 
ment, LXXIII  (May,  1927),  301, 
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II.  Interest  and  enthusiasm  in  his  subject. 

III.  Ability  to  arouse  interest  in  students. 

IV.  Organization  of  course. 

V.  Scholarship. 

VI.  Ability  to  express  thought, 

VII.  Enunciation. 

VIII.  Thinking  demanded  of  students. 

IX.  Assignments. 

X.  Leading  discussion  and  questioning. 

XI.  Sense  of  proportion. 

XII.  Feeling  between  instructor  and  students. 

XIII.  Sense  of  humor. 

XIV.  Self-confidence. 

XV.  Tolerance  and  liberality. 

XVI.  Personal  appearance. 

XVII.  Personal  peculiarities. 

While  it  is  true  that  a  few  instructors  at  Iowa  State 
College  have  complained  to  the  writer  that  a  certain  item,  or 
items,  considered  by  them  as  being  essential,  was  not  included 
in  the  scale,  none  of  the  instructors  has  suggested  that  any 
one  of  the  seventeen  items  in  the  list  above  should  be  omitted. 

Closely  related  to  the  question  of  the  particular  items 
to  be  included  in  a  rating  scale,  is  the  choice  of  the  form  or 
type  of  scale  to  use.     While  conclusive  scientific  evidence 
concerning  the  relative  virtues  of  the  different  types  is  lack- 
ing, there  seems  to  be  a  preponderance  of  opinion,  among  stu- 
dents of  rating,  in  favor  of  the  "graphic"  form.    This  has 
already  been  discussed  fully  in  Chapter  I,   (See  pages  14-18). 

2,  Administering  the  Scale. 

The  procedure  which  was  followed  in  administering  the 
scale  was  planned  very  carefully  in  the  beginning  of  this  in- 
vestigation and  has  stood  well  the  test  of  several  applications 
of  the  scale.     The  distinctive  characteristics  of  this  proce- 
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dure  were  as  follows: 

First,  the  attempt  was  made  to  word  the  instructions 
on  the  scale  so  clearly  and  definitely,  that  no  supplementary 
explanation  or  directions  v/ould  be  called  for  "by  the  raters. 

Second,  the  instructors  were  provided  with  written  in- 
structions covering  the  administration  of  the  scale.  These 
instructions  contained  a  statement  to  be  read  verbatim.  (See 
Appendix  A,  Exhibit  3)    The  instructors  were  requested  to  read 
this  statement  and  to  make  no  further  comment. 

Third,  the  instructor  left  the  room  while  the  rating 
was  being  done,  having  appointed  a  member  of  the  class  to  col- 
lect the  scales  and  to  forward  or  carry  them  to  the  office  of 
the  Council  on  Teaching. 

Fourth,  the  instructions  on  the  scale  directed  the 
student  to  refrain  from  signing  his  name  and  from  making  any 
mark  which  would  serve  to  identify  him.    He  was  also  instructed 
to  place  the  scale  in  the  envelope  provided  for  him,  and  to 
seal  it  before  passing  it  in  to  the  student  appointed  to  col- 
lect it. 

Fifth,  the  envelopes  were  opened,  the  enclosed  scales 
graded,  and  the  scores  averaged  by  regularly  appointed  persons 
working  under  the  direction  of  the  investigator.    Special  care 
was  taken  to  impress  upon  these  workers  the  necessity  of  keep- 
ing secret  any  information  gleaned  from  -che  scales  of  any  one 
or  all  of  the  instructors  rated. 

By  these  provisions  and  precautions  the  students  were 
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given  ample  assurance  that  their  individual  ratings  would 
never  become  known  to  their  instructors,  nor,  in  fact,  to  any- 
one else.     The  fact  that  the  instructor  never  saw  the  scales 
after  they  were  distributed  to  the  students  also  eliminated 
any  chance  of  the  identification  of  the  student  through  his 
v/riting  or  other  peculiarities.     The  entire  absence  of  any 
trouble  between  students  and  instructors  as  a  result  of  the  use 
of  the  scale  is  evidence  that  the  precautions  vie  have  described 
have  functioned  as  desired. 

3.  The  Computation  of  Results. 

To  facilitate  the  statistical  v/ork  involved  in  the  com- 
putation of  the  obtained  scores,  a  system  of  coding  was  designed 
and  used  throughout  the  study.     This  was  done  on  the  advice  and 
with  the  assistance  of  the  statistical  service  of  the  college. 
Instead  of  using  percentages,  ranging  from  100  to  0,  for  the 
different  positions  of  the  scale,  the  line  representing  the 
full  range  of  each  trait  was  divided  into  ten  equal  spaces. 
These  spaces  were  designated  by  integers  running  from  0  to  9 
inclusive,  with  the  zero  representing  the  space  at  the  highest 
end  of  the  scale  and  the  9,  the  last  space  at  the  low  end. 
Since  the  majority  of  the  scores  are  registered  on  the  upper 
end  of  the  scale,  where  the  divisions  are  designated  by  the 
smaller  digits,  the  labor  involved  in  the  computations  of  the 
results  was  greatly  facilitated.    Small  dots,  as  inconspicuous 

as  possible,  but  still  large  enough  to  be  seen,  were  placed 
along  the  upper  side  of  each  line,  dividing  it  into  ten  equal 
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parts.     These  dots  enabled  the  person  who  graded  the  scales  to 
determine  the  numerical  value  of  the  position  of  the  checkraark 
on  each  line  very  readily.     In  grading  the  scales,  this  value 
was  placed  at  the  right  hand  end  of  the  line  in  the  margin  of 
the  scale.     The  values  for  each  trait  were  added  together  and 
the  sum  readily  translated,  at  first,  by  means  of  a  chart  and 
later  in  the  investigation  by  the  use  of  a  set  of  tables,  into 
percentages.     It  was  thought  best,  by  the  Council  on  Teaching, 
to  report  the  scores  to  the  instructors  in  term  of  percentages, 
rather  than  in  coded  values.     These  coded  values  were  also  used 
in  the  computation  of  the  standard  deviations,  the  decoding 
being  done  by  multiplying  the  SD*s  obtained  from  the  coded 
values  by  10. 

In  some  of  the  more  special  phases  of  this  investiga- 
tion, it  was  deemed  advisable  to  use  finer  divisions  of  the 
scale,  and  for  such  phases  a  narrow  piece  of  cardboard,  the 
exact  length  of  the  lines  on  the  scale,  and  with  one  edge 
divided  into  100  equal  parts,  ?/as  designed.     This  measure  couIjJ 
be  placed  on  each  line  in  turn  and  the  exact  value  of  the  posi- 
tion of  the  check  mark  read  from  it  quickly  and  correctly.  A 
drawing  of  this  device  is  shov/n  below  in  Figure  1. 
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The  optimum  number  of  divisions  or  gradations  on  a 
rating  scale  seems  to  be  an  open  question.'^    It  is  argued  by 
certain  students  of  rating  that  the  limitations  of  human  judg- 
ment make  fine  divisions  of  a  scale  unnecessary.    According  to 
Symonds,  who  reports  the  findings  of  an  elaborate  statistical 
treatment  of  the  question,  based  upon  the  reliability  of  the 
scale,  seven  is  the  optimum  number  of  gradations. ^    Fryer  claims 
that  "in  the  rating  scale  using  objectively  defined  traits, 
there  cannot  be  more  than  five  degrees  of  proficiency  for  any 
one  trait  and  the  indication  is  that  three  degrees  of  ability 
would  lend  the  greatest  accuracy. Remmers,  on  the  other 
hand,  holds  that  because  of  the  absence  of  definite  experimental 
evidence  as  "to  how  precise  student  judgment  can  be,  it  is  best 
to  provide  the  opportunity  of  making  the  scale  sufficiently 
fine  for  all  purposes,"*^    In  the  Purdue  Scale  of  v/hich  Remmers 
is  co-author,  the  number  of  gradations  or  divisions  is  100.  In 
view  of  the  existing  disagreement  and  because  of  the  absence  of 
any  convincing  evidence,  no  great  objection  can  be  raised  a- 
gainst  the  ten  divisions  used  in  the  grading  and  statistical 
treatment  of  the  scale  developed  in  this  present  investigation. 

4.  Techniques  Used  in  Determining  Reliability. 

Two  general  techniques  were  employed  in  the  determina- 

%*ercival  M.  Symonds,  "On  the  Loss  of  Reliability  in  Ratings  due 
to  Coarseness  of  the  Scale,"  Journal  of  Experimental  Psych- 
ology,  VII  (December,  1924),  456 » 

gibid.  p.  460. 

^Douglas  Fryer,  Op.  cit.  p.  502 » 
H.  H.  Remmers,  Op.  cit.,  p.  11, 
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tion  of  the  reliability  of  the  College  Rating  Scale.     One  of 
these  is  the  technique  knov/n  as  self -correlation,®  in  which 
the  scores  obtained  in  one  application  of  the  scale  or  test 
are  correlated  with  the  corresponding  scores  obtained  in  a 
second  application  upon  the  same  subjects.     In  the  second 
method  the  differences  between  the  scores  given  to  the  same  in- 
structors at  different  times  are  employed  as  a  measure  of  the 
reliability  of  the  scale.     This  second  technique  will  be  re- 
ferred to  in  this  dissertation  as  the  "difference"  method.  It 
is  recommended  by  McCall^  as  being  superior  to  the  self-correla- 
tion method,  in  certain  respects.     Its  particular  advantages 
will  be  discussed  later  on  in  this  chapter. 

The  self -correlation  method  is  the  one  most  coimionly 
employed  in  the  determination  of  reliability  of  tests  and 
scales.     Its  technique  consists  in  calculating  the  coefficient 
of  correlation  between  the  scores  obtained  by  one  application 
of  the  test  or  scale  to  a  given  group  and  the  scores  obtained 
by  a  second  application  of  the  same  or  duplicate  test  or  scale 
to  the  same  group. 1^    In  this  particular  study  the  technique 
was  applied  as  follows:     The  investigator  made  arrangements 

%Ienry  E.  Garrett,  Statistics  in  Psychology  and  Education. 
New  York:  Longmans  Green  and  Co.     1926.     pp. 268-69, 
V/illiam  A.  McCall,  How  to  Measure  in  Education.    New  York: 

The  Macmillan  Co.     1923.     pp.  309-10.. 
Arthur  S.  Otis,  Statistical  Method  in  Educational  Measurement. 
Yonkers-on-Hudson,  New  York:  Y/orld  Book  Co.  1925.     pp. 227-28, 
^?/illiara  A.  McCall,  How  to  Experiment  in  Education.    New  York: 

The  Macmillan  Co.     1930.     pp.  112-13. 
Henry  E.  Garrett,  Op.  cit.  pp. 268-69. 

7/illiam  A.  McCall,  How  to  Measure  in  Education.    Nev;  York: 
The  Macmillan  Go.     192^3.  p.  310. 
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with  seventeen  instructors,  permitting  him  to  visit  one,  each, 
of  tr.eir  classes  and  administer  the  rating  scale.    Upon  the  day 
set,  the  investigator  entered  the  classroom  at  the  time  agreed 
upon.    He  was  introduced  to  the  class  by  the  instructor,  who 
assured  the  members  of  the  class  that  the  investigator  was 
present  with  his — the  instructor's — full  consent.    After  this 
introduction  the  instructor  left  the  room.     The  investigator 
then  proceeded  to  assure  the  students  that  the  instructor  would 
never  see  the  individual  rating  scales  and  that  every  effort 
would  be  made  to  conceal  the  identity  of  the  raters.    Each  stu- 
dent was  next  assigned  a  number  and  instructed  to  place  it  upon 
the  upper  right  hand  corner  of  the  scale,  which  was  given  to 
him  at  this  stage  of  the  proceedings.     The  students  were  then 
directed  to  proceed  with  the  rating  of  their  instructor,  accord- 
ing to  the  directions  on  the  scale.    After  completing  their 
rating  each  student  folded  his  scale  and  sealed  it  in  the  en- 
velope supplied  to  him.    No  mention  was  made  at  this  time  of  a 
subsequent  rating.     In  two  weeks  another  visit  was  paid  to  the 
class,  vjhen  the  same  procedure  was  carried  out  and  a  second 
rating  secured.     These  two  ratings  were  made  during  the  last 
three  weeks  of  the  term,  in  order  to  allow  the  students  ample 
time  in  which  to  become  well  acquainted  v/ith  their  respective 
instructors.     In  this  way  two  ratings  were  secured  from  429 
students  in  17  classes.     The  two  sets  of  scores  thus  obtained 

were  correlated  to  obtain  a  measure  of  the  degree  of  correspond- 
ence between  them.     The  coefficients  of  correlation  v/ere  ob- 
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tained  for  each  of  the  different  classes  and  the  coefficient  of 
correlation  between  the  scores  for  the  whole  429  students  was 
obtained  by  averaging-  the  coefficients  for  the  different  classes. 
In  this  computation  the  formula  for  averaging  coefficients  of 
correlation,  derived  by  R,  A.  Fisher"^"^,  was  used. 


terrnining  reliability  should  be  mentioned  at  this  time,  al- 
though the  more  pronounced  manifestations  of  it  occur  in  that 
phase  of  the  study  which  deals  with  the  validity  of  the  scale. 
(See  Chapter  VIII) .  7/hen  there  is  little  variation  betv/een  the 
individual  scores  in  each  of  the  two  variables  being  corre- 
lated, the  obtained  coefficient  of  correlation  fails  to  repre- 
sent accurately  the  amount  of  correspondence  or  coincidence 
existing  between  the  two  variables.    The  rea.son  for  this  fail- 
ure may  be  explained  by  en  analysis  of  the  product-moment 
formula,  which  is  the  formula  most  commonl]'-  used  in  simple 
correlation  work,     A  simple  form,  of  the  produce-m.oment  formula 
is:     r  =  in  which    s  summation,  x  s  deviations  from 


the  mean  of  the  scores  in  the  first  variable,  y  ss  deviations 


the  respective  stajidard  deviations  of  the  two  variables  and  N  « 
the  number  of  observations.      It  be  observed  that  the  for- 

mula, is  based  upon  the  deviations  of  the  individual  scores  from 

11  Rv  Av  Fisher,  Statistical  liethods  for  Research  V/orkers. 
(Third  Edition")     London,  Englrmd:     Oliver  and  Boyd  1930 


A  serious  limitation  of  the  correlation  method  for  de- 


from  the  meeji  of  the  scores  of  the  second  variable,  3D^  and  SD. 


pp.  177,  183-187. 


-56- 


their  mean.    If  the  individual  scores  in  each  variable  differ 
little,  or  not  at  all,  in  size,  the  deviations  from  their 
respective  means  will  he  very  small  or  be  zero.    Hence  the  ob- 
tained coefficient  of  correlation  will  be  very  smpll  or  be 
zero,  althoup;h  there  may  exist  considerable  correspondence  be- 
tween the  two  variables. 


Another  condition  may  exist  when  the  correlation  may  be 


relativels'"  high  but  the  similarity  or  correspondence  betvreen 
the  two  variables  may  be  very  slight.     This  ma.y  be  best  shoTiTi 
by  the  following  hypothetical,  but  possible,  example,  in  Y;hich 
the  two  arrays  represent  two  sets  of  scores  received  by  the 
same  teacher. 

Scores  on  first  rating  Scores  on  second  rating 


perfect,  being  represented  by  a  coefficient  of  1.00,    But  by 
the  first  of  these  tvro  sets  of  scores  the  teacher  is  placed 
among  the  lowest-rating  teachers  found  in  our  study,  while  the 
second  places  him  among  the  average,  v/e  wish,  in  this 

study  of  the  reliability  of  student  judgment,  is  a  measure  of 
how  closely  the  two  ratings,  made  at  different  times,  corre- 
spond.    It  is  very  obvious  from  the  foregoing  discussion  that 


55 
65 
75 
60 
67 
45 
50 
80 


75 
85 
95 
80 
87 
65 
70 
100 


The  correlation  between  these  two  sets  of  scores  is 
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conditions  may  exist  when  the  numerical  value  of  the  co- 
efficient of  correlation  does  not  provide  an  accurate  measure 
of  the  extent  of  this  correspondence.     These  conditions  ex- 
isted with  respect  to  sone  of  the  data  secured  in  this  inves- 
tigation.    The  recognition  of  these  limitations  led  to  the  use 
of  the  "difference"  method,  in  which  a  comparison  is  made  be- 
tween the  two  sets  of  scores  by  calculating  the  differences 
between  the  scores  of  each  pair  of  observations,  together  with 
the  means  and  distributions  of  these  obtained  differences.  To 
obtain  these  differences  the  score  for  each  trait  in  the  first 
rating  was  subtracted  from  the  score  for  the  corresponding 
trait  in  the  second  rating.     The  numerical  value  of  the  stan- 
dard deviation  of  the  individual  differences,  thus  obtained, 
from  their  mean  is  a  measure  of  the  spread  of  these  individual 
differences.-'-^    A  large  standard  deviation,  when  considered  in 
relation  to  the  size  of  the  mean,  indicates  the  existence  of 
considerable  variation  in  the  obtained  differences  betv/een  the 
two  ratings,  even  when  the  mean  of  the  differences  may  itself 
be  relatively  small.     It  is  essential,  therefore,   to  observe 
the  standard  deviations  of  the  differences  from  their  respec- 
tive means,  when  interpreting  the  obtained  differences  in  terms 
of  the  amount  of  correspondence  between  the  two  sets  of  scores 
being  compared  .-^'^ 

I^Henry  E.  Garrett,  Op.  cit.  pp.  26-27. 

William  A.  McCall,  How  to  Measure  in  Education.  New  York: 
The  Macmillan  Co.     1930.     pp. 114-115, 
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When  the  differences  between  the  scores  obtained  in 
two  ratings  are  not  great  it  is  safe  to  assume  that  the  scores 
obtained  in  the  two  ratings  correspond  closely.     There  is,  of 
course,   the  question  as  to  how  the  terra,   "not  great,"  should  be 
interpreted.     The  differences  obtained  may  be  compared  v^ith  the 
highest  possible  difference,  -.vhich,  in  turn,  would  depend  upon 
the  scale  used.     The  range  of  possible  scores  in  the  College 
Rating  Scale  runs  from  0  to  100.     The  obtained  differences  may, 
therefore,  be  stated  in  terms  of  percentages  of  the  possible 
difference  or  range.     This  affords,  at  least,  a  numerical 
standard  of  reference  by  which  to  evaluate  the  significance  of 
the  numerical  values  of  the  obtained  differences. 

In  this  investigation  the  writer  used  the  "difference" 
method  to  supplement  the  method  of  self-correlation,  not  only 
in  the  determination  of  the  reliability  of  the  College  Rating 
Scale,  in  connection  with  which  it  has  been  described  at  this 
point,  but  also  in  other  phases  of  this  study  where  its  use 
seemed  desirable.     The  detailed  treatment  given  to  it  here 
makes  it  unnecessary  to  describe  it  again  when  the  special 
techniques  employed  in  the  treatment  of  the  other  phases  are 
being  discussed. 

5.  Techniques  Used  in  the  Determination  of  Validity. 

The  technique  employed  in  the  determination  of  the 
validity  of  the  College  Rating  Scale  consisted  of  three  main 
parts:    (1)  The  careful  recording  of  observations,  made  by  the 
investigator,  in  four  consecutive  meetings  of  the  classes  of 
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each  of  27  instructors;    (2)  the  evaluation  or  grading  of  the 
instruction,  described  by  these  recorded  observations,  by 
three  recognized  authorities  on  educational  method,  v/orking  in- 
dependently of  one  another;   (3)  comparison  of  the  mean  scores 
given  to  each  instructor  by  the  three  judges,  with  those  given 
to  the  same  instructor  b;,'  the  students  in  the  observed  classes. 

The  observation  referred  to  in  (1)  above,  was  done,  in 
all  cases,  by  the  writer  who  made  an  earnest  attempt  to  enter 
the  classrooms  with  a  mind  as  free  from  prejudice  as  possible. 
It  v;as  not  the  purpose  to  criticize  the  teaching  observed,  but 
rather  to  record,  as  fully  as  could  be  done,  everything  of  sig- 
nificance which  transpired  during  each  class  meeting.     It  was 
found  possible  to  copy  verbatim  a  considerable  part  of  the  dis- 
cussions, and  special  attempts  were  made  to  record  the  main 
questions  asked  by  the  teacher,  along  with  the  students*  re- 
sponses.    In  the  records  of  these  observations  (See  Appendix  B 
Exhibit    2   ) ,  the  portions  enclosed  in  quotation  marks  are 
exactly  as  stated  in  the  classrooms. 

Immediately  following  the  observation  of  each  class 
meeting,  while  the  memory  of  what  had  occurred  v/as  still  fresh, 
the  observer  rev/rote  very  carefully  the  notes  he  had  made.  Ob- 
servations were  made  and  recorded,  in  this  manner,  of  four  con- 
secutive class  meetings  for  each  of  the  27  instructors. 

In  order  to  secure  some  indication  of  the  extent  to 
which  the  observations,   thus  recorded,  enabled  one  to  visualize 
what  transpired  in  the  classrooms,  copies  of  the  records  made 
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in  the  first  few  classes  observed,  were  subraitted  to  five  or 
six  college  instructors,  aiaong  whom  were  some  of  those  whose 
classes  had  been  observed.     There  v/as  general  agreement  that 
the  recorded  observations  enabled  a  reader  to  gain  a  rather 
clear  idea  of  what  had  transpired  in  the  classroom. 

The  instructors  were  requested  to  submit  any  syllabi, 
bibliographies,  names  of  text  and  reference  books,  copies  of 
examinations,  etc.,  which  v/ere  attached  to  the  records  of  the 
observations.     These  records  of  the  observations,  exclusive  of 
the  supplementary  material  referred  to  directly  above,  amounted 
to  494  typewritten  sheets. 

The  observations  having  been  completed,  the  next  step 
consisted  in  securing  competent  critics  or  judges  to  read  the 
recorded  observations  and  to  evaluate  the  quality  of  the  in- 
struction described  in  them.     It  was  the  intention,  at  first, 
to  secure  a  large  number  of  judges  in  order  to  insure  a  high 
reliability  for  the  mean  of  their  evaluations,  but  it  was  found 
that  very  few  of  those  approached  v/ished  to  undertake  the  task. 
Finally  the  cooperation  of  three  competent  persons  v/as  secured. 
The  three  were:     V/.  F.  Stev/art,  Professor  and  Head  of  the  De- 
partment of  Agricultural  Education,  Ohio  State  University; 

R.  Alexander,  Professor  of  Education,  the  College  of  Agri- 
culture and  Mechanic  Arts,  of  Texas;  and  Carsie  Haiiiraonds , 
Professor  of  Education,  University  of  Kentucky.     All  these  three 

educators  have  had  extended  experience  in  the  training  of  teach- 
ers and  are  specialists  in  the  field  of  methods.     In  evaluating 
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tlie  quality  of  the  instruction  described  in  the  recorded  ob- 
servations, these  three  judges  worked  independently  of  one 
another,  each  being  ignorant  of  the  identity  of  the  other  two. 

iVhen  the  records  of  the  observations  were  submitted  to 
the  judges,  a  copy  of  the  rating  scale  was  attached  to  the 
records,  v/hich  described  the  teaching  of  each  instructor.  On 
this  rating  scale  was  checked  the  raean  score,  given  to  the 
instructor  by  the  class  observed,  on  each  of  the  last  six  items 
on  the  scale.     These  items  are  of  the  nature  of  personality 
traits  which,  although  they  no  doubt  influence  the  effective- 
ness of  the  instruction,  are  not  susceptible  of  being  dis- 
covered and  evaluated  from  a  reading  of  the  observations.  It 
T/as  believed  that  the  rating  on  these  personal  traits,  given 
to  each  instructor  by  his  students,  would  aid  the  judges  in 
evaluating  more  accurately  the  effectiveness  of  the  instruc- 
tion recorded. 

The  judges  were  asked  to  place  a  checkmark  on  the  scale 
at  that  point  along  each  line,  which  represented  best  their 
judgment  of  the  quality  of  the  recorded  instruction.     It  was 
not  considered  feasible  for  the  judges  to  evaluate  the  scholar- 
ship (Trait  V)  an:;  the  enunciation  (Trait  VII)  of  the  respec- 
tive instructors  from  a  reading  of  the  records  and  they  were, 
therefore,  requested  to  omit  scores  on  these  two  items.  These 
omissions  reduced  the  number  of  items,  checked  by  the  judges, 
to  nine. 

A  critical  evaluation  of  the  techniques  described 
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above  discloses  three  major  weaknesses  or  possible  sources  of 
error.     The  first  concerns  the  method  of  observing  and  record- 
ing the  instructional  activities  in  the  classes  visited.  Did 
the  investigator  observe  and  record  everything  of  significance 
which  transpired  or  only  those  things  v/hich  his  particular 
training  and  experience  in  the  teaching  of  educational  method 
led  him  to  observe  and  to  regard  as  important?    The  observer, 
who  was  in  every  case  the  v/riter,  made  a  careful  and  delibera- 
tive effort  to  be  as  unprejudiced  and  open-minded  as  possible. 
His  own  belief  is  that  he  was  quite  successful  in  the  effort, 
but  proof  of  this  is  lacking. 

The  small  number  of  judges  or  critics,  and  the  question 
of  their  qualifications,  constitute  the  second  vulnerable  point 
in  the  teclmique.     The  average  judgment  of  many  judges  is  more 
reliable  than  that  of  a  single  one and  it  is  generally 
recommended  that  as  many  judges  or  raters  as  possible  should  be 
used.-^^    Rugg-^^  holds  that,  assuming  qualified  raters,  the  re- 
liability of  an  average  judgment  increases  directly  with  the 
square  of  the  number  of  judgments.    According  to  Rugg  the 
Probable  Error  of  a  single  judgriient  is  0.6745  SD;  of  two  judg- 
ments it  is  0,47  SD;  of  three  judgments,  0.38  SD  and  of  four 
judgments,  0.34  SD.     It  v;ill  be  noted  that  after  the  third 

■^%arold  0.  Rugg,   "Is  the  Rating  of  Human  Character  Practicable" 

Journal  of  Educational  Psychology,  XIII   (Feb.,  1922)  p.  83* 
-,c-G.  B.  Y/atson,  Op.  cit.,  p.  73. 
:^^D.  G.  Patterson,  Op.  cit.,  p.  85. 
^%arold  0.  Rugg,  Op.  cit.,  p.  83 > 
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judgment  the  reliability  of  the  average  increases  slov/ly. 
This  is  appreciated  by  Ruggl'^  when  he  writes:  "This  v/ould  in- 
sure that  practically  all  the  averages  of  three  or  four  judg- 
ments would  locate  a  person  within  his  proper  fifth  of  the 
rating  scale."    Moore^^  recoroiPxends  using  the  average  of  the 
judgments  of  three  persons  if  it  is  possible  to  get  them. 

In  this  study  the  number  of  judges  might  have  been  in- 
creased had  it  not  been  for  the  rather  laborious  task  required 
of  them,  which  made  it  difficult  to  secure  the  services  of  any 
large  number  of  competent  persons. 

The  qualifications  of  the  three  judges  selected,  may 
be  spoken  of  more  surely.     Their  respective  positions,  in  a 
definite  manner,  offer  good  evidence  of  their  professional 
knowledge  and  skill.     They  are  recognized  as  experts  in  educa- 
tional methods  by  their  fellow  v/orkers  and  appear  frequently 
on  the  programs  of  the  annual  meetings  of  local  and  national 
educational  associations. 

The  third  possible  source  of  error  in  this  technique  is 
suggested  by  the  question  as  to  how  accurately  a  competent 
critic  of  educational  method  can  determine  the  quality  of  in- 
struction from  a  written  record  or  description  of  it.  Educa- 
tional literature  abounds  with  statements  emphasizing  the  in- 
fluence of  the  personality  of  the  instructor  upon  the  effec- 

l^bid.  p.  83. 

-^*^ruce  V.  Moore,   "Personnel  Selection  of  Graduate  Engineers," 
Psychological  Monographs,  Vol.  XXX  (1921),  p.  23. 
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tiveness  of  his  instruction,  and  the  existence  and  importance 
of  the  personal  reaction  between  instructor  and  students  seem 
to  be  universally  appreciated.  It  is  probable  that  this  per- 
sonal reaction  between  instructor  and  student,  which  deter- 
mines so  largely  the  atmosphere  of  the  classroom,  is  inade- 
quately described  by  even  a  carefully  written  record  of  class 
meetings . 

An  attempt  was  made  by  this  investigator  to  present 
some  indication  of  this  factor  to  the  judges  by  giving  them 
the  mean  scores,  which  each  instructor  received  from  his  class, 
on  the  purely  personal  qualities  on  the  scale.     No  great  claim 
is  made  for  the  value  of  this  inform.ation  to  the  judges. 

Taken  together  these  three  possible  sources  of  error 
constitute  a  serious  limitation  of  the  technique  employed  in 
the  determination  of  the  validity  of  the  scale.    At  that  it 
seemed  to  this  investigator  to  possess  certain  virtues  and  in 
the  absence  of  any  technique  of  proven  worth,  it  v/as  employed 
in  this  investigation. 

Regarding  the  statistical  treatment  given  to  the  data 
derived  by  the  technique  described  above,  little  more  need  be 
said.     The  scores  obtained  are  submitted  in  detail  in  Chapter  IX. 
The  scores  given  by  the  judges  and  those  given  by  the  students 
are  compared  by  the  tv/o  methods  used  in  determining  the  reliabil- 
ity, i.e.,  the  correlation  method  and  the  method  of  differences. 
The  same  weakness  of  zhe  correlation  method,  which  has  been 
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pointed  out  in  the  section  imnediately  preceding,  is  observed 
in  even  more  pronounced  form  in  the  study  of  validity.  An 
inspection  of  a  few  of  the  data  shows  this  very  clearly.  Be- 
low in  Table  II  are  shown  the  two  sets  of  scores  received  by 
instructors  6,  8,  and  17,  v/ith  the  mean  differences  and  the 
coefficient  of  correlation  between  each  of  the  two  sets  of 
scores  for  each  instructor. 


TABLE  II 

C0L1PARIS0N  BETWEEN  THE  MEAN  SCORES  GIVEN  TO  THREE 
INSTRUCTORS  BY  JUDGES  AND  BY  STUDENTS 
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43  , 

98 

1  : 

90 

94 

4 

III 

:  30 

76 

46  ; 

92 

96 

4  : 

88 

91 

3 

IV 

:  33 

82 

49  : 

96 

96 

0  : 

90 

92 

2 

VI 

:  69 

83 

14  : 

96 

91 

5  ; 

85 

93 

8 

VIII 

:  21 

70 

49  : 

94 

97 

3  : 

91 

94 

3 

IX 

:  20 

87 

67  : 

88 

92 

4  : 

55 

79 

24 

X 

:  19 

76 

57  : 

91 

98 

7  : 

90 

92 

2 

XI 

:  33 

81 

48  : 

92 

98 

6  : 

81 

93 

12 

Average 

Difference 

46.1 

3.7 

6.9 

Coefficient 
of  Correlation  .491+. 170 


.158  ±.219 


.940 ±.159 


An  inspection  of  Table  II  reveals  very  plainly  the 
peculiar  short-comings  of  the  correlation  method  when  employed 
as  a  measure  of  the  similarity  or  correspondence  between  two 
sets  of  scores.     In  the  case  of  Instructor  No.  6  the  mean 


-66- 


difference  between  the  two  sets  of  scores  is  seen  to  be  very 
large,  i.e.,  46.1,  while  the  coefficient  of  correlation  between 
the  two  sets  is  .491±.170.     In  the  case  of  Instructor  No.  8 
the  mean  difference  between  the  two  sets  of  scores  is  3.7, 
which  indicates  a  very  high  degree  of  correspondence  between 
them,  as  may  also  be  seen  by  inspection,  but  the  coefficient  of 
correlation  between  them  is  only  .158^1.219.    For  Instructor 
No.  17  the  mean  difference  between  the  two  sets  of  scores  is 
6.9,  while  the  coefficient  of  correlation  between  them  is 
.940:t.l59.     It  is  obvious  that  the  coefficient  of  correlation 
when  applied  to  our  data  does  not  furnish  an  accurate  measure 
of  the  extent  to  which  the  judgments  of  the  students  and  of 
the  judges  coincide  or  agree.     It  would  seem  that  the  difference 
between  the  two  sets  of  scores  constitutes  a  more  accurate 
measure  of  this  agreement  than  does  the  correlation  method. 

6.  Techniques  Employed  in  the  Study  of  the  Spread  of 
General  Estimate  or  "Halo  Effect". 

The  technique  commonly  used  in  the  determination  of  the 
existence  and  extent  of  the  spread  of  general  estimate,  common- 
ly called  "halo  effect",  consists  in  correlating  the  score 
given  on  one  trait  with  the  corresponding  score  given  on  a 
second  trait.    By  this  technique  the  intercorrelations  between 
the  scores  on  the  different  traits  are  obtained.    The  numerical 
values  of  these  obtained  intercorrelation  coefficients  are 
held-^^  to  be  a  measure  of  the  extent  of  the  operation  of  "halo 

l^Russell  L.  C.  Butsch,  "Teacher  Rating,"  Review  of  Educational 
Research  Vol.  I  (April,  1931),  107.  Washington,  D.C.: 
American  Educational  Research  Association. 
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effect",  but  there  exists  no  standard  as  to  how  large  these 
coefficients  must  be  in  order  to  be  conclusive  objective  evi- 
dence of  the  existence  of  "halo", 20    Some  investigators^^  seem 
to  interpret  any  positive  coefficient  of  intercorrelation 

greater  than  zero  as  evidence  of  the  existence  of  this  factor. 

22 

Others  claim  that  one  might  well  expect  to  find  the  members  of 
such  a  carefully  chosen  and  well  trained  population,  as  is 
represented  by  a  college  faculty,  possessing  to  a  well-marked 
degree  those  qualities  generally  considered  essential  to  success 
in  the  profession.     In  other  words,  they  hold  that  such  desirable 
traits  would  be  self-supporting  and  that  a  significant  positive 
correlation  might  exist  between  them  without  indicating  the  op- 
eration of  "halo  effect".    This  investigator  failed  to  find  any 
serious  refutation  of  this,  as  a  possibility.    Disagreement  oc- 
curs when  the  attempt  is  made  to  interpret  the  obtained  inter- 
correlation coefficients  in  terms  of  the  amount  of  "halo  effect". 

These  different  methods  of  interpretation  explain  the 
discrepancy  in  the  conclusions  reached  by  the  different  investi- 
gators of  this  problem,  since  an  inspection  of  their  data  dis- 
closes little  difference  in  the  numerical  values  of  the  inter- 
correlations  obtained  by  them.    A  report  of  these  investiga- 
tions has  been  made  in  Chapter  I. 

SOsarah  E.  Marsh  and  F.  A.  C.  Perrin,  "An  Experimental  Study  of 
Rating  Scale  Technique,"  Journal  of  Abnormal  and  Social 
Psychology.  XIX  (January-March,  1925),  399. 
E.  L.  Thorndi'cce,  "A  Constant  Error  in  Psychological  Ratings," 
Journal  of  Applied  Psychology,  IV  (March,  1920),  25-29. 

H.  H.  Remmers,  Op.  cit.,  21. 
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The  intercorrelation  coefficients  obtained  in  this  in- 
vestigation were  all  positive,  and  when  corrected  for  attenua- 
tion, averaged  .47  (See  Chapter  IX,      page  152).    As  indicated 
above, coefficients  of  this  size  would  be  regarded  by  some  in- 
vestigators as  proof  conclusive  of  the  existence  of  a  large 
amount  of  "halo  effect",  while  others  would  deny  the  validity 
of  such  evidence. 

Another  technique  for  measuring  the  existence  and  ex- 
tent of  the  spread  of  general  estimate  was  employed  by  the 
present  investigator.    The  lowest  mean  score  reported  on  each 
of  the  466  classes  in  the  winter  1929  survey  was  subtracted 
from  the  highest  mean  score  for  the  same  class.    The  differ- 
ences thus  obtained  were  regarded  as  measures  of  the  extent  to 
which  the  respective  classes  discriminated  betv/een  the  various 
traits  when  rating  their  instructors.    By  referring  to  Table  XXXIV 
page  156,  it  will  be  seen  that  in  a  considerable  nuraber  of  the 
classes  the  differences  between  the  highest  and  the  lowest  mean 
scores  were  large  enough  to  suggest  rather  strongly  that,  at 
least  in  these  classes,  the  students  had  not  allowed  their  gen- 
eral reaction  toward  their  respective  instructors  to  prevent 
them  from  exercising  discrimination  as  to  the  extent  to  which 
their  instructors  possessed  the  different  qualities  of  the 
scale. 

In  fifty  per  cent  of  the  classes  the  mean  difference 
was  over  23.38  points,  while  in  the  other  fifty  per  cent  it  was 
less  than  23.38  (See  Chapter    IX      page  156).    Yet  even  in 
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these  last  classes,  the  existence  of  small  differences  is  not 
conclusive  proof  that  they  are  caused  by  the  operation  of  "halo 
effect",  since^  as  has  been  pointed  out,  it  could  very  well  be 
that  a  considerable  proportion  of  the  faculty  members  in  an 
institution,  such  as  Iowa  State  College,  would  be  rather  well- 
rounded  individuals  and  so  score  high  on  all  the  traits.  It 
might  also  be  reasoned  that  the  median  difference  of  approxi- 
mately 25  indicates  the  absence  of  "halo  effect",  on  the  ground 
that  the  average  instructor  would  not  be  likely  to  score  high- 
er than  95  on  any  one  trait  nor  lower  than  70  on  any  other  one 
of  the  traits  on  the  scale.     The  difference  therefore  that  one 
might  expect  to  find  betv/een  the  highest  and  the  lowest  scores 
would  be  in  the  vicinity  of  25. 

This  reasoning  may  not  be  sound,  but  at  least  one  can 
safely  conclude  from  the  existence  of  a  significant  proportion 
of  the  classes  in  which  the  obtained  differences  between  the 
highest  and  lowest  scores  are  over  50  points     (See  Chapter  IX 
page  156)  that  the  scale  does  not  prevent  students  from  dis- 
criminating between  the  different  traits.     It  is  evident  that, 
at  least  in  these  particular  classes,  the  students  did  exercise 
some  measure  of  discrimination.     On  the  other  hand  one  cannot 
conclude  that  discrimination  was  not  exercised  in  the  classes 
where  the  differences  betv/een  the  highest  and  lowest  mean  scores 
were  considerably  less. 

The  investigator  has  no  evidence  that  this  method  of 
measuring  the  existence  and  extent  of  the  spread  of  general 
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estimate  possesses  any  superiority  over  the  intercorrelation 
method,  since  there  is  lacking  for  it,  also,  any  objective 
standard  as  to  how  large  the  difference  "between  the  high  and 
the  low  trait  must  be  in  order  to  present  satisfactory  evi- 
dence of  the  existence  of  this  factor.     It  may  be  observed 
that  the  method  which  employs  the  difference  betv/een  the  high- 
est and  lov>^est  traits  involves  an  almost  negligible  amount  of 
labor  as  compared  with  the  intercorrelation  method.     The  writer 
is  also  unable  to  discern  any  serious  short  comings  in  the 
method. 

It  is  true  that  in  this  phase  of  the  study  the  mean 
scores,  derived  by  averaging  the  individual  scores  given  by 
the  members  of  each  class,  were  used,  while  in  the  correlation 
method  the  scores  given  by  individual  students  were  correlated. 
The  difference  method  could  be  used  with  the  scores  given  by 
individual  students  as  well  as  with  the  mean  class  scores. 

7.  Techniques  Employed  in  the  Four  Minor  Problems 
(See  Chapter  II,  page  2^ 

The  techniq.ues  employed  in  the  solution  of  these  four 
minor  problems  involved  no  principles  additional  to  those  which 
have  been  discussed  in  this  chapter.    A  detailed  discussion  of 
them  at  this  point  would  seem  to  be  unnecessary. 


CHAPTER  V 

FINDINGS  OF  THE  FIRST  GENERAL  RATING 


In  Chapter  III,  the  reader  has  already  been  informed 
that  the  first  general  application  of  the  College  Rating  Scale, 
other  than  a  few  trial  attempts  in  the  writer's  own  classes, 
was  made  in  the  fall  term  of  1928.     In  this  survey,  mimeo- 
graphed copies  of  the  scale  were  used.     (See  Appendix  A), 
This  was  regarded  as  a  preliminary  survey  from  which  it  was 
hoped  that  suggestions  for  further  revision  of  the  scale  might 
emerge.    For  this  reason  the  findings  of  this  application  of 
the  scale  v/ere  given  statistical  treatment  and  critical  study 
in  the  attempt  to  disclose  any  weaknesses  which  might  be  over- 
come by  changing  the  form  and  wording  of  the  scale,  or  the 
technique  of  its  administration. 

Two  such  weaknesses  were  apparent:   (1)  There  was  ob- 
served a  tendency  on  the  part  of  a  few  students,  in  a  consider- 
able proportion  of  the  classes,  to  place  the  check  mark  at 
exactly  the  same  place  on  each  line  of  the  scale.     In  other 
words,  the  check  marks  on  each  line  of  any  scale  so  marked, 
were  placed  directly  below  one  another,  thus  recording  exactly 
the  same  score  on  each  of  the  sixteen  traits.     The  experimenter 
regarded  this  as  evidence  that  the  students,  who  marked  the 
scales  in  this  manner,  had  failed  to  exercise  critical  judgment, 
either  because  they  did  not  take  the  task  seriously,  or  because 
they  did  not  understand  v/hat  was  required  of  them.    An  attempt 
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was  made  in  the  revised  edition  of  the  scale  to  make  such 
changes,  in  the  form  and  wording  of  the  scale  and  in  instruc- 
tions for  administering  it,  as  would  effect  a  reduction  in  the 
number  of  scales  so  marked.     (2)  What  appeared  to  be  a  more 
serious  defect  was  the  tendency  on  the  part  of  a  few  students 
to  place  the  check  marks  on  every  line  of  the  scale,  directly 
below  the  check  mark  on  the  specimen  line,  which  was  included 
in  the  instructions  at  the  top  of  the  scale.     It  seemed  clear 
that  this  line,  the  purpose  of  which  was  to  clarify  the  in- 
structions, became  for  some  students  a  cause  for  confusion. 
In  the  revision  of  the  scale  this  line  was  omitted  and  the  in- 
structions were  reworded  in  an  attempt  to  make  them  sufficient- 
ly clear  without  its  use. 

The  following  tables  present,  in  considerable  detail, 
the  findings  of  the  statistical  treatment  given  the  data  se- 
cured in  this  first  general  rating.     In  order  to  avoid  the  risk 
of  losing  the  good  will  of  the  instructors,  by  allowing  their 
ratings  to  become  common  property,  through  the  use  of  their 
names  in  the  numerous  records  of  this  study,  a  simple  coding 
system  was  adopted.    A  key  to  the  code  is  not  essential  to  the 
interpretation  of  these  data  and  will  be  omitted.     The  code  is 
used  in  Table  III  merely  for  the  purpose  of  identifying  the  in- 
dividual instructors  with  the  classes  which  rated  them.  This 
enables  the  reader  to  compare  the  scores  given  to  the  same  in- 
structor by  different  classes. 


-73- 


TABLE  III 

MEAN  SCORES  RECEIVED  BY  49  INSTRUCTORS  FROM  1654 
STUDENTS  IN  94  CLASSES,  FALL,  1928 
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1-2-1 

6 

90,60 

86,84 

55-95 

75-95 

28 

85.81 

35-95 

75-95 

11 

87.43 

35-95 

85-95 

IV-8-2 

18 

86.86 

86,86 

5-95 

75-95 

1-9-6 

12 

80.94 

79.56 

35-95 

65-95 

15 

78.19 

35-95 

65-95 

1-2-7 

11 

87.90 

90,26 

5-95 

75-95 

11 

92.62 

45-95 

75-95 

1-8-3 

21 

66.82 

68.65 

5-95 

45-95 

7 

74.18 

5-95 

75-95 

9 

68.62 

5-85 

65-95 

14 

70.06 

5-95 

55-95 

1-8-2 

14 

88.68 

88.68 

55-95 

75-95 

1-2-5 

10 

93.18 

90.61 

35-95 

85-95 

16 

89.00 

15-95 

85-95 

1-5-5 

8 

87,87 

89.28 

35-95 

85-95 

3 

92,62 

55-95 

95-95 

14 

89.37 

25-95 

75-95 

1-5-3 

35 

86.50 

86.50 

5-95 

75-95 

IY-12-3 

6 

84.31 

79.14 

25-95 

85-95 

17 

77.31 

5-95 

75-95 

IV-12-1 

13 

87.62 

86.24 

45-95 

75-95 

59 

85.94 

15-95 

75-95 

1-1-2 

23 

81.31 

81.31 

25-95 

75-95 

1-2-6 

15 

90.12 

87.32 

35-95 

85-95 

8 

82.06 

25-95 

75-95 

1-4-2 

18 

84.81 

84.81 

35-95 

75-95 

I-lO-l 

19 

83.81 

85.17 

25-95 

75-95 

9 

87.12 

5-85 

75-95 

7 

85.37 

45-95 

85-95 

1-9-7 

23 

79.56 

79.56 

25-95 

55-95 

1-5-4 

20 

89.25 

89.89 

35-95 

75-95 

13 

90.87 

55-95 

75-95 

1-3-1 

12 

93.56 

93.92 

55-95 

85-95 

21 

94.12 

65-95 

85-95 
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TABLE  III  (continued) 


Instruc- 
tor *s  Code 
Number 

Number  of 
Students 
in  Class 

Mean  Score 
by  Each 
Class 

Y/eighted 
Mean  Score 
From  All 
Classes 

Greatest 
Range  per 
Trait  in 
Each  Class 

Smallest 
Range  per 
Trait  in 
Each  Class 

11-4-1 

11 

if,  4o 

/0-90 

7 

80  .70 

ob-9o 

8o-9o 

11 

86  •  68 

00-90 

8o-9o 

I-l-o 

18 

79 . 06 

79  ♦  06 

0-9o 

o  cr    o  cr 

70-95 

1-0-1 

9 

96  .18 

90.  67 

00-90 

c\  cr    c\  cr 

9o-9o 

14 

87 . 12 

2o-9o 

o  cr  cr 

95-95 

1-4-1 

0 

93.  63 

92.60 

00-95 

o  cr    pi  cr 

80-95 

20 

94. 18 

00-90 

o  cr  cr 

85-95 

2o 

91 . 00 

25-95 

o  cr     p\  cr 

85-95 

1-7-1 

19 

85 . 93 

88,02 

5-95 

O  rr  cr 

85-95 

19 

89 . 00 

35-95 

85-95 

11 

89 . 93 

25-95 

O  cr      pi  rr 

85-95 

IV-lS-5 

19 

89 .87 

90,42 

35-95 

rj  cr     p\  cr 

75-95 

10 

90. 00 

25-95 

n  c  rr" 

75-95 

A  C 

45 

90.75 

T  rr  C 

15-95 

PC     p  rr 

65-95 

II-3-1 

43 

86 , 00 

86 . 00 

5-95 

rr  rr  rr 

55-95 

1-1-4 

19 

83.31 

83.49 

f— •  rr 

5-95 

75-95 

11 

83.81 

15-95 

85-95 

I-S-4 

13 

87 . 00 

87  .57 

25-95 

85-95 

8 

88.50 

45-95 

85-95 

1-2-3 

30 

77 . 43 

77  .43 

5-95 

75-95 

III-3-1 

31 

86.18 

87  .30 

15-95 

75-95 

19 

89.12 

15-95 

75-95 

I-l-l 

16 

77  .31 

77 . 31 

5-95 

75-95 

1-1-6 

11 

90.12 

91.16 

45-95 

85-95 

25 

91.62 

45-95 

85-95 

IV-12-2 

31 

84. 63 

84. 17 

5-95 

75-95 

8 

82.37 

35-95 

85-95 

1-2-1 

5 

92.06 

92.34 

15-95 

95-95 

7 

90.75 

45-95 

85-95 

8 

93.91 

65-95 

85-95 

1-1-5 

25 

84.93 

83.98 

35-95 

75-95 

3 

76.04 

25-95 

85-95 

1-8-1 

12 

87.12 

85.09 

45-95 

75-95 

21 

83.93 

15-95 

75-95 

IV-8-1 

14 

92.56 

90.18 

45-95 

85-95 

13 

87.62 

45-95 

75-95 

1-10-2 

24 

87.50 

85.57 

25-95 

85-95 

13 

82.00 

15-95 

75-95 

II-3-1 

33 

91.68 

91.68 

45-95 

75-95 

IV-12-4 

27 

82.77 

82.77 

15-85 

75-85 
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TABLE  III  (continued) 
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IY-12-7 

15 

73.68 

78.59 

15-95 

75-95 

40 

80.43 

25-95 

65-95 

1-5-2 

19 

79.93 

79.93 

15-95 

75-95 

IY-12-6 

49 

83.06 

82.08 

5-95 

75-95 

8 

76.05 

15-95 

75-95 

II-5-1 

12 

81.75 

81.75 

25-95 

85-95 

1-9-10 

34 

86.00 

86.00 

5-95 

75-95 

1-9-1 

8 

94.18 

92.31 

55-95 

85-95 

16 

91.37 

25-95 

75-95 

1-9-2 

19 

86.37 

89.01 

25-95 

75-95 

11 

93.56 

45-95 

85-95 

1-9-3 

16 

83.13 

86.02 

45-95 

75-95 

23 

87.87 

45-95 

75-95 

6 

86.62 

55-95 

85-95 

1-9-5 

18 

63.31 

63.31 

5-95 

75-95 

1-9-6 

12 

80.94 

79.41 

35-95 

65-95 

15 

78.19 

35-95 

65-95 

1-9-7 

23 

79.56 

79.56 

25-95 

55-95 

1-9-4 

12 

83.62 

85.79 

5-95 

85-95 

23 

83.81 

15-95 

65-95 

17 

90.00 

35-95 

95-95 

This  table  should  be  interpreted  as  follows:     The  first 
instructor,  1-2-1,  was  rated  by  three  classes  containing  6,  8, 
and  11  students  respectively.    The  mean  score  received  from  the 
first  class  was  90.60;  from  the  second,  85.81;  and  from  the 
third,  87.43.     The  weighted  mean,  or  the  mean  score  given  by 
all  the  students  in  the  three  classes  was  87.94.     The  greatest 
range  of  scores  per  trait  is  from  35  to  95  and  the  smallest 
range  is  from  85  to  95. 

When  the  table  is  studied  in  this  fashion  it  is  noted 
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that  there  is  considerable  similarity  between  the  mean  scores 
given  to  any  given  instructor,  by  his  different  classes.  In 
other  words,  no  instructor  is  given  a  low  rating  by  one  class 
and  a  high  rating  by  another.     The  differences  which  do  exist 
between  the  mean  scores  of  the  classes  of  any  one  instructor 
might  well  be  expected,  since  it  is  a  common  experience  of 
teachers  to  find  their  effectiveness  vary  noticeably  with  their 
different  classes.     This  similarity  between  the  mean  scores  re- 
ceived by  each  of  the  individual  instructors  from  his  different 
classes  was  regarded  by  the  investigator  as  an  indication  that 
the  scale  possessed  some  degree  of  reliability. 

The  mean  class  scores  range  from  66.82,  in  the  case  of 
one  class  of  instructor  1-8-3,  to  94,18  for  one  of  the  classes 
of  instructor  1-4-1.    This  range  may  be  taken  as  evidence  that 
the  students  using  the  scale  are  led  to  exercise  some  degree  of 
discrimination  in  their  judgment,  for  it  must  be  expected  that 
in  a  group  of  49  teachers  there  v/ould  be  a  considerable  range 
of  teaching  ability  and  personal  fitness  represented.     It  will 
also  be  noted  that  the  distribution  of  the  mean  class  scores 
bears  a  rough  resemblance  to  the  normal  probability  curve. 
This  is  more  clearly  shown  when  the  scores  are  cast  into  a 
frequency  distribution  as  in  Table  IV. 

^Harold  0.  Rugg,  Statistical  Method  Applied  to  Education. 
Boston:  Houghton  Mifflin  Co.     1917.  p.  153. 

Any  standard  work  on  statistics  discusses  the  characteris- 
tics of  tile  normal  probability  curve. 
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TABLE  IV 


THE  DISTRIBUTION  OF  THE  MEAI-T  SCORES  RECEIVED 
BY  49  INSTRUCTORS  FROM  94  CLASSES,  FALL,  1928 


Range  of  Scores 

Number 

of  Instructors 

Number  of 

Classes 

in 

Each  Range 

in  Each 

Range 

95-97 

0 

1 

92-94 

4 

12 

89-91 

10 

19 

86-88 

10 

22 

83-85 

9 

16 

80-82 

5 

8 

77-79 

8 

8 

74-76 

0 

3 

71-73 

0 

1 

68-70 

1 

2 

65-67 

1 

1 

62-64 

I 

1 

N 

m  49 

N  =  94 

As  might  be  expected,  the  two  distributions,  one  based 
upon  the  mean  scores  for  the  instructors  and  the  other  upon  the 
means  of  the  ratings  by  the  different  classes,  are  quite  simi- 
lar.   Both  distributions  are  considerably  skewed  toward  the 
upper  end.     A  normal  distribution  could  scarcely  be  expected 
from  this  relatively  small  sample, 2  made  up  of  instructors 
selected  largely  because  of  their  expressed  interest  in  teach- 
ing and  willingness  to  aid  in  the  investigation. 

The  correspondence  betv;een  the  scores  on  the  first  nine 
items  of  the  scale — teaching  abilities — and  the  last  seven 
items — personal  traits — is  shown  in  Table  V,  where  the  means  on 
the  two  groups  for  one  class  of  each  instructor  are  given. 


%arold  0.  Rugg,  Op.  cit.,  p.  212. 
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TABLE  V 


SHOWING  SEPARATELY  THE  MEM  SCORES  ON  NINE  TEACHING 
ABILITIES  A^TD  SEVEN  PERSONAL  TRAITS  FOR  EACH 
INSTRUCTOR  IN  THE  PRELBIINARY  SURVEY,  FALL,  1928 


tjO  1 
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O  -H  ^ 
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CO  CO  +^ 

W  ^^  CQ  03 

CO 

as  -P 

CO  ?H  CQ  00 

u 

Q)  -H  +^  W 

(D  CQ 

0  -H  +J  CQ 

©  -P  CQ 

•p 

P^  -H  +3  g 

-p 

F<  rH  CQ  S 

Ph  -H  -P  ^ 

m 

CO        T-i  H  ^ 

•H  F^  © 

CO          CD   (/J  © 

©     ^<  cd  -p 

<D 

J3  -H  -P 

©  fJ  Fh  Cd  -p 

M 

g  O  <i1  P«H  M 

S  O  Eh  1-1  M 

M 

S  O  <1  M 

S  O  (H  f-J  M 

1-2-1 

88.44 

92.14 

IV-8-2 

85.77 

87.00 

1-2-7 

87.77 

87.28 

1-8-3 

61.55 

73.57 

1-8-2 

89.77 

88.42 

1-2-5 

92.88 

93.57 

1-5-5 

85.88 

90.42 

1-5-3 

85.66 

87.57 

IV-12-3 

80.00 

89.85 

IV-12-1 

85.88 

89.85 

1-1-2 

79.89 

83.14 

1-2-6 

87.00 

95.57 

1-4-2 

84.11 

85.71 

I-lO-l 

OO  .  44 

84.28 

1-5-4 

89.  66 

88.71 

1-3-1 

93.00 

94.28 

II-4-1 

76.00 

79.28 

1-1-3 

74.44 

86.14 

1-5-1 

94.77 

98.00 

1-4-1 

94.55 

92.42 

1-7-1 

81.66 

91.42 

IV-12-5 

89.55 

92.57 

II-3-1 

82.55 

90.43 

1-1-4 

79.66 

88.00 

1-2-4 

84.55 

90.14 

1-2-3 

75.00 

80.57 

III-3-1 

87.33 

84.71 

I-l-l 

77.00 

85.71 

1-1-6 

90. 55 

89.57 

IV-12-2 

83.33 

86 . 28 

1-2-1 

89.00 

96.00 

1-1-5 

82.00 

88.71 

1-8-1 

87.11 

87.14 

IV-8-1 

90.33 

95.42 

1-10-2 

86.88 

88.28 

II-3-1 

90.77 

92.85 

IV-12-4 

32.55 

83.00 

IV-12-7 

71.88 

76.00 

1-5-2 

78.22 

82.14 

IV-12-6 

82.00 

84.42 

II-5-1 

79.00 

85.28 

1-9-10 

84.55 

87.85 

1-9-1 

94.55 

93.71 

1-9-2 

84.44  • 

88.85 

1-9-3 

81.78 

84.85 

1-9-4 

61.89 

65.15 

1-9-5 

81.33 

80.43 

1-9-6 

82.22 

85.43 

1-9-7 

77.44 

82.28 

Average 

83.66 

87.23 

Table  V  is  to  be  read  as  follows: — Instructor  1-2-1  re- 
ceived a  mean  score  of  88,44  on  the  first  9  items  and  a  mean 
score  of  92,14  on  the  last  7  items.     The  data  for  the  other  in- 
structors are  read  in  the  same  manner. 


The  obtained  coefficient  of  correlation  between  the  mean 
scores  on  the  two  groups  of  items  is  .84i.033.     This  high'^  cor- 
relation is  not  surprising,  since  the  influence  of  a  teacher's 
personality  upon  the  effectiveness  of  his  instruction  is  quite 
generally  recognized  by  students  of  the  problem.^  The  fact  that 
the  scale  registered  this  correlation  between  personality  and 
effectiveness  of  instruction  encouraged  the  investigator  in 
this  first  attempt. 

Up  to  this  point  the  mean  scores  on  the  v/hole  seven- 


%enry  E.  Garrett,  Statistics  in  Psychology  and  Education.  New 
York:  Longmans  Green  and  Co.     1926.     p.  170. 

Guy  M.  Wilson  and  Kremer  J,  Hoke,  How  to  Measure.    New  York: 
The  Macmillan  Go.     1988.     p.  536. 

William  A.  McCall,  How  to  Measure  in  Education.    New  York: 
^    The  Macmillan  Co,     1923.     p.  393. 

A.  C.  Boyce , "Qualities  of  Merit  in  Secondary  School  Teachers," 
Journal  of  Educational  Psychology,  III  (March,  1912),  144-157 

F.  B.  Knight,  Qualities  Related  to  Success  in  Teaching.  Con- 
tributions to  Education,  No,  120.    New  York:  Teachers  Col- 
lege, Columbia  University.  1922. 

Arthur  C.  Boyce.  Methods  for  Measuring  Teachers*  Efficiency. 
Fourteenth  Yearbook,  Part  II,  National  Society  for  the  Study 
of  Education.    Bloomington,  Illinois:  Public  School  Publish- 
ing Co.     1915.  pp.  29-41. 

W.  F.  Book,  "The  High  School  Teacher  from  the  Pupil's  Point  of 
View,"    Pedagogical  Seminary,  XII,  239-88. 

Wr,  C.  Rudeiger  and  George  D.  Strayer,  "The  Qualities  of  Merit 
in  Teachers,"    Journal  of  Educational  Psychology,  I  (May, 
1910),  272-78. 

W.  Vv.  Charters  and  Douglas  Waples,  The  Commonwealth  Teacher 
Training  Study.     Chicago:  University  of  Chicago  Press.  1929. 
pp.  14-19. 

J.  H.  Bradley,  "A  Study  of  the  Relative  Importance  of  the 
Qualities  of  a  Teacher  and  Her  Teaching  in  Their  Relation 
to  General  Merit,"  Educational  Administration  and  Supervi- 
sion, IV  (September,  1918),  358-63. 

Charles  Fordyce,   "Notes  on  the  Correlations  between  General 
Teaching  Power  and  Some  Specific  Teaching  Qualities.  Eight- 
eenth Yearbook,  Part  I.  National  Society  for  the  Study  of 
Education.     Bloomington,  Illinois:  Public  School  Publishing 
Co.,  1919,   pp.  349-51. 
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teen  items  of  the  scale  and  on  the  two  groups  of  items,  i.e., 
the  teaching  abilities  and  the  personality  traits  for  each  in- 
structor have  been  compared,    A  study  of  the  mean  for  each  of 
the  items  on  the  scale  was  next  made.    Table  VI  presents  the 
significant  findings  of  this  phase  of  the  study.    The  mean  re- 
corded for  each  trait  is  the  average  of  all  the  scores  given  by 
the  1654  students,  on  that  trait;  the  standard  deviations  of 
the  distribution  and  the  probable  errors  of  the  means  are  based 
upon  the  individual  scores  on  each  trait,  given  by  the  1654 
students. 


TABLE  VI 

MEAl^S  mi)  DISTRIBUTIONS  OF  THE  SCORES  RECEIVED 
BY  49  INSTRUCTORS  FROM  1654  STUDENTS,  FALL,  1928 


Mean    SD  PE 


Traits 

dist . 

M 

I. 

Preparation  for  class  meetings 

85.95 

14.91 

.247 

II. 

Interest  and  enthusiasm  in  his 

subject 

87.22 

13.98 

.231 

III. 

Ability  to  arouse  interest  in 

students 

78.40 

21.50 

.356 

IV. 

Feeling  between  instructor  and 

students 

87.76 

16.26 

.269 

V. 

Organization  of  course 

83.87 

16.82 

.278 

VI. 

Thinking  demanded  of  students 

79.19 

19.64 

.325 

VII. 

Ability  to  express  thought 

83.52 

16.67 

.276 

VIII. 

Enunciation 

90.32 

13.44 

.222 

IX. 

Leading  discussion  and  questioning 

81.48 

19.11 

.316 

X. 

Personal  appearance 

92.44 

10.16 

.168 

XI. 

Personal  peculiarities 

89.16 

13.70 

.227 

XII. 

Interest  in  students 

85.86 

15.66 

.219 

XIII. 

Sense  of  proportion 

87.36 

14.58 

.241 

XIV. 

Sense  of  humor 

83.78 

17.43 

.289 

XV. 

Self-confidence 

85.96 

16.60 

.273 

XVI. 

Tolerance  and  liberality 

89.39 

14.84 

.246 
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This  table  should  be  read  as  follows;  The  mean  of  all 
the  1654  scores  on  Trait  I — Preparation  for  Class  Meetings — is 
85.95;  sixty-eight  per  cent  of  the  scores  fall  between  85.95 
14.91,  or  between  71.04  and  100;  and  the  chances  are  50  in  100 
that  the  obtained  mean  of  85.95  does  not  diverge  from  the  true 
mean  by  more  than  .247.     This  last  is  a  significant  figure  since 
it  measures  the  reliability  of  the  mean  obtained  from  the  1654 
students.     If  scores  should  be  received  from  each  of  several 
other  groups  of  students  for  the  same  instructors  50  per  cent 
of  the  means  thus  obtained  on  Trait  I  would  fall  between  85.95 
il.247  or  between  85.70  and  86.19.    We  can  be  practically  cer- 
tain that  the  true  mean  lies  within  the  limits  85.95l!4  x  .247, 
or  between  86.938  and  84.962.    The  table  is  read  in  the  same 
manner  for  the  other  traits. 

In  interpreting  the  significance  of  the  standard  devia- 
tion in  Table  YI  it  should  be  kept  in  mind  that  the  obtained 
mean  for  any  given  trait  is  the  mean  of  all  the  scores  received 
by  49  instructors  from  the  1654  students.     In  the  case  of  most 
of  the  traits  the  standard  deviation  from  the  mean  is  snail 
enough  to  indicate  considerable  agreement  between  the  raters. 
However,  in  revising  the  scale  an  attempt  was  made  to  reduce 
the  standard  deviations  by  making  the  descriptive  statements, 
under  the  lines,  more  specific  and  objective,  and  by  other 
changes  suggested  earlier  in  this  chapter,  especially  for  those 
traits  which  shov;ed  the  largest  standard  deviation. 

The  inter-correlations  between  the  different  items  on 
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the  scale  v/as  calculated  in  order  to  determine  (1)  the  existence 
and  extent  of  the  tendency  of  the  raters  to  allow  their  gener- 
al reaction  toward  their  instructors  to  influence  the  ratings 
on  specific  traits;   (2)  the  existence  of  duplication  among  the 
traits,  which  adds  to  the  number  of  the  items  on  the  scale 
without  contributing  much  additional  information  concerning  the 
person  being  rated.    The  numerical  values  of  the  coefficients 
of  correlation  obtained  when  different  traits  are  correlated 
one  with  the  other,  in  pairs,  are  generally  regarded  as  mea- 
sures of  both  the  conditions  referred  to  above.     This  has  al- 
ready been  discussed  fully  in  Chapter  IV  (See  pages  56-70  ). 

In  Table  VII  the  coefficients  of  correlation  obtained 
from  the  scores  of  this  first  survey,  fall  1928,  are  presented. 

Table  VII  should  be  read  as  follows:     The  coefficient 
of  correlation  between  Trait  I — Preparation  for  Class  Meetings — 
and  Trait  II — Interest  and  Enthusiasm  in  his  Subject — is  .491 
.013.     If  other  surveys  were  made  with  similar  groups  of  stu- 
dents the  chances  are  50  in  100  that  the  obtained  coefficients 
would  fall  betv/een  .491  ±.013,  or  between  .504  and  .478.  V/e 
can  be  practically  certain  that  the  true  coefficient  lies  v;ith- 
in  .491±4  X  .013,  or  between  ,543  and  .439.     The  data  for  the 
other  comparisons  should  be  read  in  the  same  v/ay. 

In  every  case  the  probable  error  is  small  enough  to  in- 
dicate a  high  reliability^ for  the  obtained  coefficients  of  cor- 

^Arthur  S,  Otis,  Statistical  Method  in  Educational  Measurement. 
Yoxmkers-on-Hudson,  New  York:  V/orld  Book  Co.     1925.     p.  256. 
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TABLE  YII 


INTERCORRSLATIONS  BETVVEElNf  TRAITS,  BASED  UPON 
THE  IffiAN  GLASS  SCORES  GIVEN  BY  94  CLASSES 
TO  49  TEACHERS,  FALL,  1928 
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V 

.443 

.013 
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.015 
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XI 
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.423 

.013 
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.576 

.011 

XI 

XIII 

.352 

.014 

III  - 

Y 
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.012 

XI 

XIV 

.404 

.014 

III  - 

VI 

.412 

.014 

XI 

XV 

.374 

.014 

III  - 

VII 

.507 

.012 

XI 
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.014 
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.346 

.014 
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.014 
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V 

.359 

.014 

XII 

XIV 

.426 

.013 
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VI 

.297 

.015 

XII 

XV 

.354 

.014 
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.363 

.014 

XII 

XVI 

.477 

.013 

IV  - 
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.285 

.015 

XIII 

XIV 

.379 

.014 
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VI 

.326 

.015 

XIII 

XV 

.466 

.013 

V  - 

VII 

.447 

.013 

XIII 

XVI 

.351 

.014 

V  - 

VIII 

.296 

.015 

XIV 

XV 

.452 

.013 

VI  - 

VII 

.295 

.015 

XIV 

XVI 

.405 

.014 

VI  - 

VIII 

.171 

.016 

XV 

XVI 

.398 

.014 

VII  - 

VIII 

.451 

.013 

IV 

XII 

.610 

.010 

IX  - 

X 

.254 

.015 

Average 

.379 

.014 

relation. 

All  possible  correlations  between  the  traits  were  not 
calculated.    To  do  so  v/ould  have  entailed  considerable  addition- 
al expense  and  a  close  inspection  of  the  scores  received  by  the 
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different  instructors  revealed  only  one  pair  which  seemed  more 
highly  correlated  than  those  referred  to  in  Table  VII.     The  co- 
efficient of  this  suspected  pair,  which  included  Trait  IV — 
Feeling  between  Instructor  and  Students — and  Trait  XII — Inter- 
est in  Students — was  found  to  be  ,61021*7.015.     It  was  decided 
to  omit  Trait  XII  in  the  revision  of  the  scale  to  be  used  in 
later  surveys.     This  was  done  because  of  the  necessity  to  keep 
the  number  of  items  in  the  scale  to  the  minimum,  and  the  high 
correlation  between  the  two  traits  in  question  indicated  that 
the  score  on  either  one  would  not  often  vary  greatly  from  the 
score  on  the  other.     Trait  XII— Interest  in  Students — was 
dropped  from  the  list  because  it  was  thought  to  be  more  subjec- 
tive and  more  easily  simulated  by  the  instructor  than  the  other 
one  of  the  pair,  namely,  Trait  IV — Feeling  between  Instructor 
and  Students. 

The  numerical  value  of  the  coefficients  of  correlation 
between  different  pairs  of  traits  is  generally  held  to  be  a 
measure  of  the  "halo  effect".    This  has  been  commented  upon  al- 
ready in  Chapter  I,  page  24  and  in  Chapter  IV,  page  66,  and  will 
receive  further  treatment  in  Chapter  VIII.    There  is  general 
agreement  among  workers  in  the  rating  field  that  these  inter- 
correlations  between  traits  should  be  small,  but  there  is  no 
agreement  as  to  hov/  large  the  coefficients  must  be  in  order  to 
serve  as  proof  of  the  existence  of  "halo  effect".     In  the  ab- 
sence of  a  definite  objective  standard  the  writer  could,  at 
best,  compare  the  coefficients  obtained  in  this  study  with 
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those  reported  by  other  investigators.    The  corresponding  data 
concerning  the  Purdue  Rating  Scale  for  Teachers  as  reported  by 
Stalmaker  and  Reimners^  permit  of  a  direct  comparison  being  made, 
since  the  Purdue  Rating  Scale  and  the  one  described  in  this 
thesis  are  designed  for  the  same  purpose  and  are  applied  under 
much  the  same  conditions.    Stalmaker  and  Remmers  claim  for  the 
Purdue  Rating  Scale  for  Teachers  a  freedom  from  "halo  effect," 
and  submit  as  evidence  intercorrelation  coefficients  distri- 
buted as  shown  in  Table  VIII,  v.hich  follows. 


TABLE  VIII 

INTERCORRELATIONS  BETWEEN  TRAITS  IN  THE  PURDUE  SCALE 
AND  IN  THE  COLLEGE  RATING  SCALE 


Numerical  Values  of 
Inter cor relations 


.01 
.11 
.21 
.31 
.41 
.51 
.61 


.10 
.20 
.30 
.40 
.50 
.60 
.70 


Number  of  Coefficients  in  Each  Interval 


Purdue  Scale     •  College  Rating  Scale 


2 
2 
9 
10 
14 
6 
2 


0 
1 
14 

20 
17 
4 
1 


N 


45 


N  =  57 


On  the  basis  of  the  numerical  values  of  the  intercorre- 
lation coefficients  presented  in  Table  VIII,  the  College  Rating 
Scale  used  in  this  investigation  compares  very  favorably  with 
the  Purdue  Rating  Scale.    The  means  of  the  intercorrelation  co- 

M.  Stalmaker  and  H.  H.  Remmers,  "Can  Students  Discriminate 
Traits  Associated  with  Success  in  Teaching?"  Journal  of  Ap- 
plied Psychology.  XII  (December,  1928),  502-10. 
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efficients  are  almost  identical,  being  .383  for  the  Purdue 
Scale  and  ,380  for  the  College  Rating  Scale. 

In  sum,  this  first  application  of  the  preliminary  Col- 
lege Rating  Scale  of  16  items  or  traits  by  1654  students  in  94 
classes,  to  the  rating  of  49  instructors,  had  netted  the  fol- 
lowing: 

1.  The  discovery  of  certain  definite  weaknesses  in  the 
scale  used  in  this  preliminary  survey.     These  weaknesses  have 
already  been  discussed  in  this  chapter.     (See  page  71 )« 

S.  Evidence  of  a  fair  degree  of  reliability  possessed 
by  the  scale,  as  displayed  in  the  similarity  of  the  ratings  re- 
ceived by  individual  instructors  from  two  or  more  of  their 
classes.     (See  Table  III,  page  73). 

3.  The  fact  that  the  scores  for  the  49  instructors 
ranged  from  66.82  to  94.18,  in  a  distribution  roughly  approxi- 
mating the  normal  curve,  may  be  regarded  as  evidence  that  the 
scale  caused  students  to  exercise  discrimination,  otherwise  the 
scores  received  by  all  the  49  instructors  should  have  been  very 
similar.     (See  Table  IV,  page  77), 

4.  Considerable  agreement  among  the  students  as  to  the 
extent  to  which  any  given  teacher  possessed  each  individual 
trait.    This  was  indicated  by  the  numerical  values  of  the  stand- 
ard deviations  shown  in  Table  YI.   (See  page  80). 

5.  The  scale  seemed  to  have  overcome,  to  some  extent, 
the  tendency  of  the  rater  to  be  unduly  influenced  by  his  gener- 
al reaction  toward  the  person  being  rated,  as  indicated  by  the 
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niimerical  values  of  the  intercorrelation  coefficients.  (See 
Table  VII,  page  83    and  Table  VIII,  page  85). 

6.  The  active  interest  displayed  by  the  instructors  in 
the  use  of  the  scale  and  in  the  scores  which  they  received 
from  its  application  in  their  classes  confirmed  the  hope  that 
the  scale  would  operate  as  a  means  of  improving  the  quality  of 
instruction  in  classes  where  it  was  applied. 


CHAPTER  VI 

THE  FINDINGS  OF  SUCCESSIVE  RATING  SURVEYS 


Table  I  of  Chapter  III  contains  a  statement  of  the 
source  of  data  for  the  present  investigation.     The  first  exten- 
sive application  of  the  preliminary  form  of  the  College  Rating 
Scale  is  reported,  in  Chapter  V.     It  confirmed  the  value  of  the 
study  and  led  to  the  revision  of  the  scale  and  its  determina- 
tion in  its  present  form.     (See  page  39.)    This  revision  of  the 
scale  is  the  one  used  in  subsequent  surveys  reported  in  this 
study.     In  this  chapter  it  is  proposed  to  present  the  more 
general  findings  of  these  eight  successive  surveys. 

At  the  close  of  the  winter  term  of  1929  (Item  2,  Table 
I),  of  the  fall  term  of  1929,   (Item  4,  Table  I),  and  of  the 
vfinter  quarter  of  1930,   (Item  5,  Table  I),  the  more  general 
findings  of  the  surveys  of  the  respective  quarters  were  pub- 
lished in  the  form  of  small  bulletins  and  v/ere  distributed  to 
members  of  the  faculty.    These  bulletins  contained  additional 
material  relative  to  the  improvements  of  certain  deficiencies 
which  the  ratings  had  disclosed.     The  same  form  of  organization 
of  the  data  was  not  used  in  these  three  reports  because  it  was 
desired,  in  each  of  the  reports  of  the  sruveys,  to  emphasize 
different  problems  connected  with  the  task  of  measuring  and  im- 
proving the  prevailing  instruction,  through  the  use  of  the 
scale.    The  surveys  made  in  the  winter  term  of  1929,  being  the 
first  and  most  extensive,  will  be  reported  in  more  detail  than 
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the  others. 

In  the  first  column  of  Table  IX      are  given  the  mean 
scores,  rounded  to  the  first  decimal  place,  received  in  the 
winter  term  of  1929,   (Item  2,  Table  I),  by  the  entire  203 
teachers  on  each  of  the  seventeen  traits  on  the  scale.  The 
second  column  gives  the  means  of  all  the  ratings  received  by- 
teachers  of  senior  classes,  while  the  third,  fourth  and  fifth 
columns  contain  the  means  for  the  scores  given  by  the  junior, 
sophomore  and  freshman  classes  respectively. 

The  mean  scores  of  the  ratings  given  by  each  of  the 
four  college  years  were  computed,  in  the  first  place,  in  order 
to  satisfy  certain  instructors  who  insisted  that  ratings  made 
by  the  classes  in  the  different  years  were  not  directly  com- 
parable with  one  another,  because  of  the  difference  in  the  ma- 
turity of  the  students.     It  will  be  observed  that'  the  differ- 
ences which  range  from  2  to  6  points  between  the  mean  scores 
for  the  different  years  are  surprisingly  small  in  the  case  of 
every  trait.    A  more  refined  treatment  of  this  problem  is  pre- 
sented in  another  chapter,   (See  page  166). 

The  close  agreement  among  the  four  classes,  as  to  the 
specific  traits  in  which  their  instructors  are  strong  or  weak, 
should  be  kept  in  mind  when  the  reliability  of  student-rating 
is  being  studied.    A  full  discussion  of  the  question  of  relia- 
bility will  be  presented  in  Chapter  VII. 

The  question  of  the  distribution  of  the  ratings  given 
by  the  students  is  no  less  important  than  that  of  the  mean 
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TABLE  IX 


MEM  SCORES,  BY  TRAIT,  GIVEN  BY  8464  STUDENTS  TO  203 
INSTRUCTORS,  AND  BY  STUDENTS  OF  THE  FOUR  DIFFERENT 
COLLEGE  YEARS  TO  THEIR  INSTRUCTORS,  WINTER,  1929 
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I.      Preparation  for  class 
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80.5 

79.0 

80,0 

80.6 

II. 

Interest  and  enthusiasm 

in  his  subject 

86.0 

86.5 

83.5 

87.0 

84.9 

III. 

Ability  to  arouse  interest 

in  students 

73.1 

74.0 

71.0 

74.5 

71.9 

IV. 

Organization  of  course 
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VI. 

Ability  to  express  thought 

HP.  n 
/  o  •  u 

/  o .  u 

/  D  .  U 

/  O  .  0 

"7  r?  n 

VII. 

Enunciation 

ox  .  o 

OO  •  y: 

PI  p\ 

o± .  o 

P9  ^7 

pri  "7 

OU  .  f 

VIII. Thinking  demanded  of 

student 

76.2 

78.7 

76.0 

75.7 

74.5 

DC. 

Assignments 

78.6 

76.5 

78.0 

80.3 

78.0 

X. 

Leading  discussion  and 

questioning 

73.5 

74.5 

75.0 

78.1 

73.1 

XI. 

Sense  of  proportion 

78.7 

78.5 

78.2 

79.0 

79.6 

XII. 

Feeling  between  instructor 

and  students 

81.5 

80.1 

80.0 

81.9 

78.0 

XIII 

.Sense  of  humor 

76.6 

77.4 

76.9 

78.0 

73.4 

XIV. 

Self-confidence 

82.2 

84.7 

79.4 

81.8 

81.7 

XV. 

Tolerance  and  liberality 

80.8 

81.7 

83.5 

82.0 

79.3 

XVI. 

Personal  appearance 

85.6 

89.0 

85.3 

85.5 

85.6 

XVII. Personal  peculiarities 

81.1 

81,4 

82.3 

81.5 

80.1 

Averages 

79.8 

80*4 

79.1 

79*9 

78i9 
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scores  in  properly  interpreting  the  findings.     In  Table  X  the 
extent  of  the  deviation  of  the  ratings  from  the  mean  for  each 
trait  is  presented  in  terms  of  the  standard  deviations  from 
the  mean. 

TABLE  X. 

STANDARD  DEVIATIONS  FROI.:  THE  MEAI^I  FOR  EACH  TRAIT  AI^ID 
FOR  THE  DIFFERENT  YEARS,  WINTER  1929. 


Traits  on  Scale 

All 
Classes 
(8464) 

Seniors 
(561) 

Juniors 
(466) 

Sopho- 
mores 
(1368) 

Fresh- 
men 
(2863) 

1  . 

jrreparauion  lor  c±ass 

meetings 

17.3 

15.3 

17.6 

16.8 

17.8 

II. 

Interest  and  enthusiasm 

in  his  subject 

13,5 

12.0 

15.9 

12.3 

14.4 

III. 

Ability  to  arouse  in- 

terest in  students 

22.4 

22.4 

24.2 

21.1 

23.0 

IV. 

Organization  of  course 

19.5 

19.7 

21.4 

17  .4 

19.4 

V. 

Scholarship 

13.9 

11.9 

16.4 

14.5 

13.9 

VI. 

Ability  to  express 

thought 

18.8 

17.9 

19.5 

18.1 

19.4 

VII. 

Enunciation 

17.7 

14.6 

16.5 

16.9 

19.1 

VIII. 

Thinking  demanded  of 

student 

20,7 

17.3 

22.1 

18.6 

20.9 

XI. 

Assignments 

19.2 

19.1 

19.0 

16.5 

20.0 

X. 

Leading  discussion  and 

questioning 

21.8 

21.4 

19.9 

16.9 

21.9 

XI. 

Sense  of  proportion 

16.8 

18.3 

20,4 

18.7 

18.6 

XII. 

Feeling  between  instruc- 

tor and  students 

20.5 

19.7 

20.3 

18.7 

22.3 

XIII. 

Sense  of  humor 

21.9 

21.1 

21.6 

19.9 

24.0 

XIV. 

Self-confidence 

16.7 

14.4 

19.4 

16.8 

17.0 

XV. 

Tolerance  and  liberality  18,4 

16.8 

14.4 

16.7 

20.5 

XVI. 

Personal  appearance 

13.4 

10.3 

12.9 

13.1 

13.9 

XVII. 

Personal  peculiarities 

18.5 

19.1 

17.4 

17.3 

19.9 

Averages 

18.4 

17.1 

18.8 

17.1 

19.2 
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Table  X  should  be  read  as  follows,  using  the  item  No. 
X  in  the  first  column  as  an  example:    Sixty-eight  per  cent  of 
the  8464  individual  scores  fall  within  17,3  points  on  either 
side  of  the  mean,  that  is,  between  97,3  and  62,7  (mean  equals 
80,0),     In  the  same  way,  58  per  cent  of  the  561  individual  scores 
on  item  No,  1,  second  column,  fall  within  15,3  on  either  side  of 
the  mean  80,5,  or  between  85,8  and  65.2.     The  table  is  read  in 
the  same  way  for  all  the  other  classes  and  items. 

The  standard  deviations  in  Table  X  indicate  the  de- 
gree of  agreement  among  the  students  as  to  the  extent  to  which 
their  instructors  possess  any  given  trait.     It  may  be  noted  that 
the  agreement  is  much  closer  in  certain  traits  than  in  others. 
Trait  ZVI--Personal  Appearance— shows  the  least  deviation,  with 
Trait  II— Interest  and  Enthusiasm  in  Subject — and  Trait 
Scholarship — rather  close  seconds.     On  the  other  hand.  Trait  III — 
Ability  to  Arouse  Interest — Trait  XIII — Sense  of  Humor — and  Trait 
X— Leading  Discussions  and  Questioning--have  a  much  greater  de- 
viation in  the  individual  scores  v/hich  go  to  make  up  the  mean  of 
each.     One  might  easily  reason  that  the  character  of  the  traits 
in  each  of  these  two  groups  of  traits,  thus  differentiated,  ex- 
plains the  difference  in  the  variability  of  student  judgment 
when  applied  to  them.    To  follow  this  reasoning  is  not,  however, 
a  part  of  the  purpose  of  this  study.     It  is  sufficient  to  note 
at  this  point  that  our  data  indicate  that  the  judgment  of  stu- 
dents on  some  of  the  items  or  traits  of  the  scale  is  much  more 
variable  than  on  others.     It  is  apparent,  also,  from  the  data 
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presented  in  Table  X,     that  there  exists  no  consistent  differ- 
ence, in  general,  between  the  standard  deviations  of  the  first 
ten  traits,  which  mi ^t  be  referred  to  as  teaching  abilities, 
and  the  last  seven  which  partake  more  of  the  character  of  per- 
sonal traits. 

The  deviations  or  spread  in  the  individual  scores  on 
each  of  the  items  in  the  scale  may  be  differentiated  also  on 
the  basis  of  the  four  college  years.     It  is  apparent  that  some 
differences  exist  here  also.     The  average  of  19.2  for  the  stan- 
dard deviations  from  the  means  given  by  freshman  classes,  in- 
dicates that  the  scores  given  by  the  freshiaen  tend  to  scatter 
more  than  do  those  of  the  other  years,  particularly  the  senior 
and  sophomore.     This  is  probably  v/hat  might  be  expected,  as  the 
maturity  of  any  individual  is  currently  held  to  be  a  function  of 
his  judgment.     The  influence  of  the  maturity  of  the  student  up- 
on his  rating  of  his  instructor  was  regarded  as  of  sufficient 
importance  to  merit  closer  investigation,  and  a  more  extended 
treatment  of  the  question  is  reported  in  Chapter 

The  Findings  of  Subsequent  Ratings 
Because  of  the  large  number  of  teachers  and  students 
involved  in  the  v/inter  1929  survey,  and  because  it  was  the  first 
and  most  extensive  survey  made  with  the  revised  scale,  the  find- 
ings have  been  set  forth  in  considerable  detail.     The  writer  does 
not  consider  it  necessary  to  report  the  findings  of  subsequent 
surveys  in  such  detail,    V/hile  the  statistical  treatment  given 
the  data  varies  somewhat  in  each  term,  in  every  case  the  means 
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of  the  scores  given  by  each  class,  along  with  the  standard 
deviations  of  the  scores  from  the  mean,  and  the  probable  er- 
rors of  the  means  have  been  computed  for  all  the  different 
surveys.     Table  XI  contains  these  data  for  the  surveys  made 
subsequent  to  that  of  the  winter,  1929.     The  findings  of  the 
winter,  1930,  survey,  were  treated  in  such  a  manner  as  to  dis- 
close the  difference  between  the  ratings  made  in  the  different 
divisions  of  the  college,  but  such  data  are  not  germane  to  the 
problems  of  this  study,  and  will  not  be  reported.     Suffice  it 
to  say  that  no  significant  differences  between  the  mean  scores 
for  each  division  were  disclosed. 

Table  XI  should  be  read  as  follows:     On  Trait  I,  Pre- 
paration for  Class  Meetings,  the  mean  score  in  the  spring,  19S9, 
survey  was  84.8;  sixty-eight  per  cent  of  the  individual  scores 
on  Trait  I  fell  between  84.8  ±  10.88  or  between  95,68  and  73.92, 
Moreover,   the  chances  are  even  that  the  obtained  mean  of  84.8 
does  not  differ  from  the  true  mean  by  more  than  — .232  and  that 
we  may  be  certain  (the  chances  are  99  to  100)  that  the  true  mean 
lies  within  the  limits  84.8 ±4  x  .232,  or  between  83.87  and 
85.72"'".     The  data  for  the  other  surveys  and  for  the  other  traits 
are  interpreted  in  the  same  manner. 

A  close  inspection  of  Table  XI  discloses  several 
rather  interesting  and  perhaps  significant  details :- 


'Henry  E.  Garrett,  Statistics  in  Psychology  and  Education. 
New  York:  Longmans,  Green  and  Co.     1926  p.l25» 
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1.  There  is  a  gradual  and  fairly  regular  increase  in 
the  means  of  the  ratings,  from  term  to  term.     One  should  be 
careful  in  interpreting  the  meaning  of  this  increase.  There 
is  an  inclination  on  the  part  of  those  who  are  interested  in 
the  use  of  the  scale  for  the  improvement  of  teaching  to  inter- 
pret this  increase  in  scores  as  indicating  increased  efficiency 
in  the  work  of  the  instructors.    But  at  least  two  factors,  af- 
fecting the  situation,  must  be  taken  into  account.     In  the 
first  place  the  ratings,  until  the  validity  of  the  scale  has 
been  established,  must  be  regarded  merely  as  a  record  of  stu- 
dent judgment  and  may,  therefore,  not  be  an  accurate  measure 

of  teaching  efficiency,  and  in  the  second  place,  it  must  be 
kept  in  mind  that  the  personnel  of  the  groups  of  instructors 
involved  in  the  different  surveys  was  continually  changing. 

This  matter  of  the  increase  in  ratings,  made  at  dif- 
ferent times,  is  treated  more  carefully  in  another  section  of 
this  study,     (See  page  178), 

2.  The  traits  on  the  scale  maintain  much  the  same 
relative  position  each  term,  with  respect  to  the  mean  rating 
given  them.    This  could  be  interpreted  as  evidence  of  the  con- 
sistency or  reliability  of  student  judgment,  insofar  as  the 
meein  ratings  given  by  large  numbers  of  students  are  concerned, 
but  more  conclusive  evidence  of  reliability  is  submitted  in 
Chapter  VII. 

The  increase  in  the  mean  scores  from  term  to  term 
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was  greater  in  the  case  of  certain  items  on  the  scale  than 
for  others.    For  example,  the  increase  in  the  mean  scores 
for  Trait  III, — Ability  to  Arouse  Interest  in  Students — , 
Trait  VIII — Thinking  demanded  of  Students — , Trait  X — ^Heading 
Discussion  and  Questioning — ,  and  Trait  XVII — Sense  of  Humor — , 
waa  11  points  each,  while  for  Trait  II— Interest  and  Enthus- 
iasm in  his  Subject--,  Trait  V — Scholarship — ,  and  Trait 
XVI — Personal  Appearance — ,  the  increase  was  much  less, being 
5,  4,  and  6  points  respectively.     It  will  be  noted  that  those 
traits  or  items,  which  gained  the  most,  were  the  lowest  at 
the  beginning  of  the  investigation.     This  fact  may  account 
for  the  large  increase  obtained  on  these  items  in  subsequent 
ratings,  as  instructors  might  be  expected  to  concentrate  on 
the  improvement  of  their  v;eakest  traits.    Moreover,  there  exist- 
ed more  room  for  improvement  in  the  cases  of  these  particular 
traits.     It  might  also  be  argued  that  the  traits  in  question 
are  the  ones  most  susceptible  of  relatively  rapid  improvement 
as  a  result  of  conscious  endeavor  on  the  part  of  the  individ- 
ual instructors. 

However  sound  these  conclusions  may  be  it  is  not 
our  purpose  to  discuss  them  at  this  point.     Those  interested 
in  the  use  of  the  scale  as  an  instrument  in  the  improvement  of 
teaching  might  well  be  tempted  to  interpret  the  increase  in 
the  scores  as  an  evidence  of  improvement  in  teaching  effective- 
ness and  to  attribute  this  improvement  to  the  influence  of  the 
the  continued  use  of  the  scale.     vVhile  such  a  conclusion  may 
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be  perfectly  sound  and  be  of  sufficient  importance  to  warrant 
further  investigation,  the  problem  will  not  be  pursued  further 
at  present.    A  more  pertinent  problem  to  the  purpose  of  this 
study  has  to  do  with  the  reliability  of  the  scale  when  used 
in  classes  of  the  normal  size.     The  presentation  of  such  ev- 
idences of  reliability  as  were  disclosed  in  the  process  of  the 
investigation    becomes,  therefore,  the  next  task. 


CHAPTER  VII 
EVIDENCES  OF  KELIiiBILITY 


The  central  problem  of  this  chapter  is  to  apply  to  the 
data  secured,  such  tests  of  reliability  as  are  available,  and  to 
reach  conclusions  concerning  the  reliability  of  the  College  Rating 
Scale.    As  previously  stated  (see  page  113)  the  reliability  of  any 
measuring  scale  refers  to  the  consistency  with  which  it  measures 
the  same  product  from  time  to  time,  in  different  applications  of 
the  scale.    The  technique  used  by  this  investigator  in  the  determi- 
nation of  reliability  has  been  fully  discussed  in  Chapter  IV.  The 
chief  feature  of  the  technique  employed  consisted  in  comparing  the 
results  obtained  by  two  applications  of  the  scale  to  the  same  groups. 

This  is  known  as  the  method  of  self-correlation  and  the  "r" 
so  foxind  is  called  the  "reliability  coefficient,"^    The  numerical 
value  of  the  coefficient  of  reliability  is  a  measure  of  the  accuracy 
with  which  the  scale  or  test  measures  whatever  it  does  measure.  It 
is  a  completely  reliable  measure  of  whatever  it  measures  when  the 
coefficient  of  reliability  is  1.00,  and  its  reliability  decreases  as 
the  coefficient  approaches  .00,  having  at  that  point  no  reliability 
whatsoever. 


Henry  E,  Garrett.    Statistics  in  Psychology  and  Education. 
New  York:    Longmans,  Green  &  Co.,  1986.  p.  317. 

Any  one  of  the  standard  works  on  the  use  of  statistics  in  education 
discusses  this  point.    A  list  of  such  references  is  to  be  found  in 
the  bibliography. 
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The  lower  the  reliability  coefficient,  the  less  is  the  relia- 
bility or  consistency  of  the  scale  or  test  as  a  measuring  in- 
strument. 

The  failure  of  the  coefficient  of  correlation  to 
measure  accurately  the  correspondence  between  two  groups  of 
variables  has  already  been  pointed  out  in  Chapter  IV.  Another 
measure  of  reliability  is  described  in  the  same  chapter.  This 
measure  makes  use  of  the  differences  between  each  pair  of  scores 
obtained  by  the  two  applications  of  the  scale.    Both  of  these 
methods  will  be  made  use  of  in  determining  the  reliability  of 
the  college  Teachers  Rating  Scale. 

The  evidences  of  the  reliability  of  the  College 
Teachers  Rating  Scale  which  this  investigation  has  disclosed, 
will  be  presented  in  the  follov/ing  order: 

1.  Comparison  and  analysis  of  the  ratings  obtained 

by  three  individual  instructors  from  a  number  of  classes  taught 
during  the  same  term. 

2.  Comparison  of  the  ratings  made  of  the  same  instruc- 
ors,  by  the  same  students,  at  different  times, 

3.  Comparison  of  the  mean  scores  received  by  a  large 
niimber  of  instructors  from  classes  taught  during  the  same  terms. 

The  statistical  data  pertaining  to  each  of  the  fore- 
going comparisons  will  now  be  presented  in  some  detail, 

1.  Comparison  and  Analysis  of  the  Ratings 
Obtained  by  Three  Individual  Instructors 
from  a  Number  of  Classes  Taught  During  the 
Same  Term, 
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Three  instructors  were  chosen  for  this  particular 
phase  of  the  study.     The  first  of  these  is  a  representative  of 
the  group  of  relatively  high-scoring  instructors,  i.e.,  those 
whose  mean  scores  hover  around  90,    The  second  is  a  represent- 
ative of  the  large  middle  group  whose  mean  scores  are  in  the 
vicinity  of  84  and  85,  and  the  third  represents  fairly  well  the 
low-scoring  group,  receiving  an  average  of  less  than  70, 

In  addition  to  the  mean  scores  on  each  trait,  the  dis- 
tribution of  the  scores  given  by  the  individual  students  on  each 
trait  was  also  considered  in  the  selection  of  these  representa- 
tive instructors.    An  inspection  of  the  report  forms  (See  Appen- 
dix A.    Exhibit  4),  which  give  the  distribution  of  the  individ- 
ual scores,  enabled  the  writer  to  select  classes  in  which  the 
distribution  of  the  scores  was  quite  typical  of  those  in  the 
three  groups.     This  was  the  basis  of  selection  in  the  case  of 
the  first  two  instructors,  but  in  the  case  of  the  third,  an  at- 
tempt was  made  to  select  one  whose  class  reports  showed  the  very 
widest  distribution  of  the  individual  scores.    This  was  done  to 
show  just  how  wide  the  distribution  of  the  scores  might  be  in 
the  case  of  any  individual  instructor. 

The  significant  statistical  data  pertaining  to  each  of 
these  instructors  are  r. resented  in  the  tables  which  follow. 

In  Table  XII    the  scores  received  by  a  representative 
of  the  high-scoring  group  of  instructors,  from  three  classes, 
taught  during  the  same  term,  are  shown  in  some  detail. 
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TABLE  XII 


COIvIPARISON  OF  IIEAN  SCORES  RECEIVED  BY  ONE  OF  THE 
HIGH-SCORING  INSTRUCTORS  FROM  THREE  CLASSES 
IN  THE  SAIiIE  COURSE  A1\l)  IN  THE  SAi;IE  TERM. 


Traits 

:  Class  A, 

N  =  29 

:  Class  B, 

K  =  24 

:  Class  C, 

N  =  14 

•  "KIT 

.  M 

PErv. 
m 

SD ,  .  J. 
dist 

:  M 

^^dist 

PE„ 

m 

SDj  4  4. 

'-'^dist 

I 

:92 

.81 

6.48 

• 

:90 

1.17 

8.52 

• 
• 

:94 

.73 

4.07 

II 

:92 

.81 

6.48 

:91 

.92 

6.77 

:94 

.73 

4.07 

III 

:89 

.97 

7.73 

:90 

.91 

6.65 

:85 

1.20 

6.65 

IV 

:89 

1.30 

10.41 

:89 

1.14 

8.30 

:95 

.47 

2.59 

V 

:90 

.70 

5.58 

:90 

1.10 

8.02 

:93 

1.05 

5.83 

VI 

:90 

1.30 

10.41 

:89 

1.08 

7.83 

:94 

.80 

4.44 

VII 

:92 

.83 

6.59 

:90 

.88 

6.40 

:95 

.47 

2,59 

VIII 

:76 

2.38 

19.02 

;83 

2.05 

15.07 

:83 

2.52 

14.00 

IX 

:88 

1.33 

10.62 

:88 

1.10 

8.80 

:87 

1.34 

7.44 

X 

:80 

3.37 

26.85 

:85 

1.78 

12.91 

:86 

1.60 

8.86 

XI 

:90 

1.02 

8.11 

;87 

1.24 

9.04 

:91 

1.12 

6.23 

XII 

:90 

.77 

6.12 

:63 

1.01 

7.31 

:78 

2.93 

16.22 

XIII 

:92 

.89 

7.12 

:91 

.96 

7.07 

:97 

1.30 

7.32 

XIV 

:89 

1.36 

10.82 

:89 

1.14 

8.30 

:94 

.93 

5.18 

XV 

:85 

1.52 

12.12 

:83 

1.48 

10.79 

:67 

5.46 

30.30 

XVI 

:91 

.81 

6.43 

:91 

.96 

7.07 

:94 

.93 

5.18 

XVII 

:87 

1.57 

12.51 

:90 

• 
• 

1.10 

8.02 

:81 

• 

2.57 

14.26 

Total 

:88.3  .35 

2.79 

• 

•88.5  .30 

• 

2.16 

•88.3 

• 

,48 

2.64 

Table  XII  should  be  read  as  follows,  using  Trait 


I  as  an  example:     The  mean  score  received  by  the  instructor 
from  29  students  in  Class  A  is  92.    Sixty-eight  per  cent  of  the 
29  individual  scores  lie  within  92  ±  6.48,  or  between  98.48  and 
85.52.     If  ratings  were  made  of  the  same  instructor  by  other 
classes  of  similar  composition  and  size,  the  chances  are  50  in 
100  that  the  means  would  fall  within  92  ±  .81,  or  between  92.81 
and  91.19,  and  it  is  practically  certain  that  the  true  mean  lies 
between  92  ±  4  x  .81,^  or  between  95.24  and  88.76.  


Ibid. pp.  125-26. 
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By  such  an  interpretation  of  the  data  for  the  other 
classes  and  items  in  Table  XII  the  reliability  of  the  mean 
scores  is  determined.    The  standard  deviations  of  the  distri- 
bution of  scores  on  each  trait,  except  for  Trait    VIII — Think- 
ing Demanded  of  Students--,  and  Trait  X — Leading  the  Class 
Discussion,  in  Classes  A  and  B,  and  for  Traits  VIII,  XII,  XV, 
and  XVII  in  Class  C,  indicate  a  rather  well-marked  concentra- 
tion of  the  individual  scores  around  the  mean  of  each  trait. 
The  probable  errors,  except  in  the  case  of  the  particular 
traits  referred  to  above,  indicate  that  the  means  of  further 
samplings  would  not  vary  greatly  from  the  obtained  means. 
The  numerical  value  of  the  probable  errors  of  the  means,  with 
the  exception  of  those  for  the  traits  mentioned  above,  lie  be- 
tween 0  and  2.     The  chances  are  even  that,  with  further  sam- 
ples, the  means  for  the  majority  of  the  traits  will  fall  within 
2  points  on  either  side  of  the  obtained  means  in  Table  XIII. 
It  will  be  noted  that  standard  deviations  and,  hence,  also  the 
probable  errors,  vary  considerable  in  size  among  the  various 
traits,  indicating  that  on  certain  traits  there  is  much  closer 
agreement  among  the  students  than  on  others.    With  some  small 
variation  the  traits  maintain  much  the  same  rank,  in  the  three 
classes,  with  respect  to  their  standard  deviations, 

A  further  measure  of  the  reliability  of  the  mean 
scores  reported  in  Table  XII  is  obtained  by  taking  the  dif- 
ferences between  the  mean  scores  of  the  three  pairs  of  classes. 


Ibid,  p,  126. 
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Table  XIII  presents  these  differences,  along  with  measures  of 
their  reliability. 

TABLE  XIII 

SIGNIFICANCE  OF  DIFFERENCES  BETWEEN  THE  ivIEAlT  SCORES 
GIVEN  BY  THREE  CLASSES  TO  THE  INSTRUCTOR 


MOSE 

RATINGS  ARE 

SHOY/N  IN  TABLE 

XII. 

Classes 

Difference 

P.  ^*^^ff 

Diff 

Chances  in 

•^•^•diff 

100 

A  -  B 

.12 

.46 

.26 

57 

B  -  C 

.12 

.56 

.21 

55 

A  -  C 

.00* 

.59 

.00 

50 

*The  means  of  class  A  and  class  C  are  identical. 

Thble  XIII  is  read  as  follows:     The  difference  between 


the  total  means  for  Class  A  and  B  is  ,12.    The  chances  are  50  in 
100  that  the  obtained  difference  of  .12  does  not  diverge  from 
the  true  difference  by  more  than  +  .46,     It  is  practically  cer- 
tain that  .12  does  not  differ  from  the  true  difference  between 
trie  means  by  more  than  4  x  .46  or  1.84.    Moreover,  there  are  57 

chances  in  100  that  the  true  difference  between  the  means  of 

4 

Class  A  and  B  is  greater  than  zero. 

The  most  striking  feature  presented  in  Table  XIV  is 
the  small  differences  which  exist  between  the  total  nsan  scores 
given  by  the  three  classes.     It  must  be  admitted  that  these  dif- 
ferences are  less  for  this  instructor  than  the  mean  difference 

^Ibid. pp. 133-36. 
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between  scores  given  by  different  classes  to  the  other  instruc- 
tors in  this  high-scoring  group,  and  cannot,  therefore,  be  re- 
garded as  typical  of  the  group  insofar  as  this  one  factor  is 
concerned.     The  differences  between  the  ratings  given  by  several 
pairs  of  classes  will  be  presented  later  in  this  chapter.  With 
regard  to  the  spread  or  distribution  of  the  individual  ratings, 
however,  these  three  classes  may  be  regarded  as  quite  represent- 
ative of  the  group,  and  it  was  mainly  with  this  matter  of  dis- 
tribution in  mind  that  the  choice  of  these  classes  was  made. 

TABLE  XIV 


COIvIPARISONS  OF  mAJ^^  SCORES  RECEIVED  BY  OlsS  INSTRUCTOR, 
OF  THE  AVERAGE-SCORING  GROUP,  FROM  THREE  CLASSES 
IN  SAIvIE  SUBJECT  AI^IC  IN  S^-.IE  TERIvI. 


Traits 

:  Class  A, 

N  s  23 

:  Cla 

ss  B, 

N  =  21 

:  Class  C, 

N  =  24 

:  M 

SDdist 

:  M 

• 
• 

^^dist 

:  M 

• 
• 

PE^ 
m 

^^dist 

I 

.88 

1.04 

7.36 

!87 

1.19 

8.12 

;87 

.91 

6.63 

II 

.90 

1.00 

7.14 

:90 

.74 

5.00 

:88 

1.01 

7.35 

III 

:76 

2.20 

15.58 

:71 

2.74 

18.61 

:74 

1.98 

14.41 

IV  ; 

:77 

2.22 

15.79 

:77 

3.01 

20.54 

:77 

2.13 

15.47 

V  ; 

:92 

.67 

4.78 

:86 

1.28 

8.73 

:87 

1.10 

8.00 

VI  : 

:85 

1.35 

9.59 

:85 

1.84 

12.53 

:86 

.99 

7.25 

VII  : 

1 91 

.70 

4.96 

:90 

.99 

6.72 

:90 

.97 

7.07 

VIII  : 

:58 

2.10 

14.90 

:63 

3.10 

21.11 

:70 

2.24 

16.29 

IX  ; 

:85 

1.65 

11.69 

:85 

1.92 

13.09 

:86 

.95 

7.02 

Z  : 

:74 

2.64 

18.73 

:75 

1.98 

13.45 

:72 

2.55 

18.51 

XI  : 

:88 

2.10 

14.82 

:87 

1.73 

11.76 

:81 

2.00 

14.69 

XII  : 

:92 

.71 

5,05 

:93 

.75 

5.27 

:87 

2.00 

14.69 

XIII  : 

:86 

1.62 

11.54 

:85 

2.04 

13.88 

:80 

1.91 

13.85 

XIV  : 

:99 

.82 

5.80 

:87 

1.28 

8.68 

:86 

1.31 

9.56 

XV 

i8Q 

1.05 

7.48 

:85 

1.79 

12.15 

:80 

2.42 

17.56 

XVI  : 

:93 

.58 

4.12 

:91 

1.07 

7.28 

:92 

.76 

5.53 

XVII  : 

:91 

.81 

5.76 

:91 

• 
• 

1.07 

7.28 

:87 

• 
• 

1.31 

9.56 

Total 

:84.9*  .47 

5.36 

• 

:84.0*  .44 

• 
• 

3.01 

:82. 

• 

9*  .41 

2.95 

*The  total  averages  are  rounded  to  the  first  decimal  place. 
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In  Tables  XIV  and  XV  are  presented  corresponding  data 
with  reference  to  another  instructor  v/ho  was  rated  by  three 
classes  in  the  same  subject  during  one  term.    This  instructor 
is  a  typical  representative  of  the  large  average-scoring  or  mid- 
dle group  of  instructors. 

Table  XIV  should  be  read  in  the  same  manner  as  indi- 
cated for  Table  XII.     The  mean  score  received  by  the  instructor 
from  Class  A  on  Trait  I  is  88.    Sixty-eight  per  cent  of  the  23 
individual  scores  on  Trait  I  lie  between  88  ±  7.36,  or  betv/een 
95.36  and  80.64.     If  ratings  v;ere  given  to  the  same  instructor 
by  other  classes  of  like  composition  and  size  the  chances  are 
50  in  100  that  their  means  would  fall  vathin  88  ±  1.04  or  be- 
tween 89,04  and  86.96  and  it  is  practically  certain  that  the 

5 

true  mean  lies  between  88  ±  4  x  1.04  or  between  9E.16  and  83.84. 

A  study  of  Table  XIV  reveals  certain  significant  facts: 
The  means  of  the  sarae  traits  in  the  three  classes  are  seen  to 
correspond  rather  closely  in  value.    The  traits  maintain  the 
same  relative  position,  with  reference  to  their  means,  in  the 
classes.    With  the  exception  of  Traits  III,  IV,  VIII,  X,  XI,  and 
XIII,  the  standard  deviations  may  be  regarded  as  indicating 
rather  well  mari^ied  concentration  of  the  scores  around  the  means. 
The  numerical  values  of  the  probable  errors  of  the  means  indi- 
cate considerable  reliability  for  the  obtained  means. 

This  reliability  is  indicated  more  clearly  in  Table 

XV  where  the  significance  of  the  differences  between  the  means 

is  presented.  

^  Ibid.  p. 84,  85,  125,  126. 
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TABLE  XV 

SIGNIFICANCE  OF  DIFFERENCES  BETWEEN  TKS  J.IEAN  SCORES 
GIVEN  BY  'THREE  CLASSES  TO  THE  INSTRUCTOR 


mOSH  RATINGS  ARE  SH0V;N  IN  TABLE  XIV. 


Classes         Difference      P.  E.^j^^f       Diff  Chances  in 

pTlTciifj.  100 

A  -  B  .94  .64  1.47  84 

A  -  C  2.000  .64  3.12  98 

B  -  C  1.06  .60  1.76  88 


This  table  is  read  in  the  same  way  as  indicated  for 
Table  XIII.    The  difference  between  the  total  mean  scores  of 
Classes  A  and  B  is  ,94.     The  chances  are  50  in  100  that  the 
obtained  difference  of  .94  does  not  diverge  from  the  true  mean 
by  more  than  .64.     It  is  practically  certain  that  .94  does  not 
differ  from  the  true  difference  between  the  means  by  more  than 
4  X  .64  or  2,56.    The  chances  are  84  in  100  that  the  true  dif- 
ference between  the  means  is  greater  than  zero.    The  remaining 
differences  are  interpreted  in  like  manner. 

An  inspection  of  Tables  XIV  and  XV  reveals  a  quite 
striking  similarity  in  the  data  for  the  three  different  classes 
compared.    The  standard  deviations  of  the  distribution  of  the 
scores  from  the  means  average  only  a  little  higher  than  those 
in  Table  XII.    For  the  majority  of  the  traits  the  standard  dev- 
iations are  small  enough  to  indicate  close  agreement  amojag  the 
raters.     While  the  chances  are  high  that  there  are  true  differ- 
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ences  between  the  means  of  the  different  classes,  it  is  also 
true  that  these  differences  are  small. 

In  Tables  XVI  and  XVII  the  corresponding  data  for  one 
of  the  low-scoring  instructors  are  presented.    These  three 
classes  were  selected  in  order  to  show  how  wide  the  distribution 
of  the  scores  may  be  in  some  cases. 

TABLE  XVI 


C01\5PARIS0NS  0?  I^AN  SGORSS  RECEIVED  BY  OI^IE  INSTRUCTOR, 
OF  THi  AVERAGE-SCORING  GROUP,  FROM  THREE  CLASSES 
IN  SAJVIE  SUBJECT  Al^^D  IN  SAim  TERM. 


Traits 

:  Class  A, 

N  =  14 

:  Class  B, 

N  =  14 

:  Class  c, 

U  «  IS 

:  M 

m 

SD  -,  .  ^ 

^^dist 

:  M 

^^dist 

:  M 

SD^  .  4. 
^^dist 

I 

:35 

5.22 

28.97 

:57 

3.83 

21 , 21 

• 
• 

:57 

3.45 

21.68 

II 

:87 

•  98 

5,45 

:91 

1.01 

5.58 

:85 

1.24 

7.81 

III 

:47 

5.06 

28.06 

:59 

4.41 

24.45 

:50 

2.77 

17.40 

IV 

:46 

4.26 

25.60 

:66 

4.49 

24.91 

:56 

3.25 

20.41 

V  ; 

:83 

2.62 

14,55 

:78 

1.37 

7.57 

:84 

2.15 

13.54 

VI  : 

:75 

2.90 

16.07 

:67 

2.04 

11.33 

:76 

2.45 

15.42 

VII  : 

67 

3.46 

19.17  ! 

:59 

3.44 

19.09 

:69 

2.74 

17.24 

VIII  : 

77 

1.60 

8.86  ' 

:85 

2.02 

11.21 

:71 

2,56 

16.07 

IX  : 

:64 

3.32 

18.42 

:81 

4.75 

26,32 

:66 

2.98 

18.70 

X  : 

:56 

3,98 

22.15 

:57 

3.67 

20.35 

:59 

2.92 

18.33 

XI  : 

.54 

4.72 

26.18  . 

:57 

5,56 

30.82 

:63 

2.86 

18.00 

XII  : 

:35 

5.44 

30.17 

:51 

4.52 

25.07 

:61 

2.60 

16.37 

XIII  : 

:57 

2.13 

11.79  : 

:62 

4.41 

24,43 

:70 

2.92 

18,33 

XIV  : 

77 

2.88 

15.98 

:79 

3.20 

17.74 

:73 

1.85 

11.56 

XV  : 

50 

4.06 

22,52  . 

:64 

4.99 

27,66 

:84 

1.35 

8,49 

XVI  : 

.86 

1.54 

8.55  : 

:89 

.96 

5.34 

:77 

1.21 

7.64 

XVII  : 

:55 

3.08 

17.08  , 

:52 

4,50 

24,93 

:66 

3.31 

20.81 

Total  : 

61.8 

*  ,89 

4.92  : 

:67.9 

*  .92 

5,08 

:68.6 

• 

*  ,63 

3.96 

The  total  averages  are  rounded  to  the  first  dedimal  place. 


Table  XVI  should  be  read  as  follows,  using  Trait  I 
in  Class  A,  as  example;     The  mean  score  received  by  the  instruc- 
tor from  Class  A  is  35.    Sixty-eight  per  cent  of  the  14  Individ- 
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ual  scores  on  Trait  I  lie  between  35  ±  28.97,  or  63.97  and 
6.03.     If  ratings  were  received  from  other  classes  of  like 
composition  and  size  the  chances  are  50  in  100  that  their 
means  would  fall  within  35  ±  5.22  or  between  40.22  and  29.78, 
and  it  is  practically  certain  that  the  true  mean  lies  between 
35  ±  4  X  5.22,  or  between  55.58  and  14.12. 

The  low  mean  scores  and  the  relatively  high  standard 
deviations  and  probable  errors  in  the  case  of  certain  of  the 
traits  constitute  the  essential  features  of  Table  XVI.  The 
high  standard  deviations  on  certain  traits  indicate  a  very  wide 
divergency  of  opinion  among  the  students  as  to  the  extent  to 
which  this  instructor  possessed  the  particular  traits  to  which, 
the  high  standard  deviations  refer.    The  large  probable  errors 
of  the  means  indicate  that  the  obtained  means  in  the  case  of 
these  same  traits  are  quite  unreliable.    However,  on  Traits 
II,  VIII,  XI,  and  XVI  the  standard, deviations  are  quite  low. 
It  should  be  kept  in  mind  here  that  these  classes  were  chosen 
because  of  the  wide  distribution  of  the  scores  in  order  to  dem- 
onstrate the  possibilities  in  this  direction.     The  variation 
in  the  mean  scores  for  the  three  classes  is  not  great,  except 
in  the  case  of  Trait  I,  and  the  traits  maintain  much  the  sarae 
relative  position,  with  regard  to  the  numerical  value  of  their 
means,  in  each  of  the  three  classes. 

The  significance  of  the  differences  between  the  means 
of  the  scores  given  by  the  three  classes  is  shown  in  Table  XVII. 
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TABLE XVII 

SIGNIFICAI^ICE  OF  DIFFERr.NCES  BET^ffiEN  THE  fffiAL^'  SCORES 
GIVEN  BY  THREE  CLiiSSES  TO  THE  INSTRUCTOR 
VffiOSE  RATINGS  ARE  SPiOV/N  IN  TABLE  XVI. 

Classes         Difference      P.  E.j^^ff       Dif f .  Chances  in 

^•^•diff.  100 

A  -  G  6.82  1.09  6.26  100 

B  -  C  .76  1.12  .68  68 

A  -  B  6.06  1.28  4.73  100 


Table  XVII  is  read  in  the  same  manner  as  indicated 
for  Tables  XIII  and  XV.     The  difference  between  the  total  mean 
scores  for  classes  A  and  C  is  6.82.     The  chances  are  50  in  100 
that  the  obtained  differences  of  6.8?:!  does  not  diverge  from  the 
true  difference  by  more  than  1.09.     It  is  practically  certain 
that  the  obtained  difference  of  6.82  does  not  differ  from  the 
true  differences  between  the  means  by  more  than  4  x  1.09  or 
4.36.      The  chances  are  100  in  100  that  the  true  difference  be- 
tween the  means  is  greater  than  zero.    The  corresponding  data 
for  the  two  other  comparisons  between  the  classes  are  inter- 
preted in  the  same  manner. 

'While  such  a  comparison  of  the  scores  received  by 
individual  instructors  from  two  or  more  classes  is  enlighten- 
ing, the  labor  involved  prohibits  the  study  of  very  many  in- 
structors in  this  detailed  manner.    A  more  extensive  attempt 
to  determine  the  reliability  of  the  scale  is  described  in  the 
next  section. 
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2.     Comparison  of  the  Ratings  Made  of  the  Same 
Instructors  by  the  Same  Students,  at  Dif- 
ferent Times. 

The  data  utilized  in  this  phase  of  the  study  are 
treated  in  such  a  way  as  to  produce  coefficients  of  correlation 
between  the  findings  of  two  applications  of  the  College  Teachers 
Rating  Scale,  made  two  weeks  apart.     The  coefficients  thus  ob- 
tained are  taken  to  be  measures  of  the  degree  of  correspondence 
between  the  findings  of  the  two  applications  of  the  scale,  and 
are  known  as  "reliability  coefficients."    In  addition  to  the 
coefficient  of  correlation  the  mean  scores  were  compared  in  an- 
other way  by  obtaining  the  differences  between  them. 

Three  distinct  coefficients  of  reliability  were  ob- 
tained by  correlating  the  data  in  three  different  ways,  as 
follows: 

a.  Correlation  between  the  mean  scores,  by  traits, 
received  by  17  instructors  from  429  students,  on 
two  ratings  made  two  weeks  apart, 

b.  Correlation  between  the  sum  of  the  scores  on  all 
the  traits,  given  by  each  of  429  students  to  his 
instructor,  at  two  ratings  made  two  weeks  apart. 

c.  Correlation  between  the  scores,  on  each  trait, 
given  by  each  of  429  students  to  his  instructor, 
at  two  ratings  made  two  weeks  apart. 

To  secure  the  data  for  these  correlations  the  inves- 
tigator made  arrangements  with  17  instructors  to  visit  their 
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classes  and  administer  the  rating  scale.    The  instructor  left 
the  room  after  introducing  the  writer  to  the  class  and  assur- 
ing its  members  that  the  investigator  was  present  with  his, 
the  instructor  *  s»  full  consent.    The  v/riter  then  assured  the 
students  that  the  instructor  would  never  see  the  individual 
rating  scales  and  that  every  precaution  would  he  taken  to 
conceal  the  identity  of  the  rater.    Each  student  was  assigned 
a  nuiaber  and  instructed  to  place  it  upon  the  upper  edge  of  the 
scale,  which  was  given  him  at  this  stage  of  the  proceedings. 
They  were  then  instructed  to  proceed  with  the  rating  of  their 
instructor  according  to  the  instructions  on  the  scale.  Each 
student  folded  the  scale  and  sealed  it  in  an  envelope  supplied 
to  him.    No  mention  was  made  of  a  subsequent  rating  at  the 
tine.     In  two  weeks  another  visit  was  paid  to  the  class,  when 
the  same  procedure  was  carried  out  and  a  rating  once  more 
secured.     These  two  ratings  were  made  during  the  last  three 
weeks  of  the  term  in  order  to  allow  the  students  araple  time  to 
become  well  acquainted  with  their  instructor.    A  more  detailed 
description  of  this  technique  is  given  in  Chapter  IV. 

In  this  way  two  ratings  were  secured  from  4S9  stu- 
dents in  17  classes.    There  were  more  students  enrolled  in 
these  classes  but  on  one  or  the  other  of  the  days  on  which  the 
ratings  were  made  several  students  v/ere  absent  and  two  ratings 
from  them  could  not  be  secured. 

The  material  thus  secured  was  used  in  the  correlations 
referred  to  at  the  beginning  of  this  section  of  the  study.  The 
data  and  conclusions  concerning  each  of  these  correlations  will 
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now  be  presented  separately  and  in  some  detail. 

a.  Correlation  betv/een  the  Mean  Scores,  by  Traits, 
Received  by  17  Instructors  from  429  Students  on 
Ratings  Made  Two  Weeks  Apart. 

In  this  correlation  the  means  of  the  scores  on  each 

trait  were  used.     The  coefficient  of  correlation  for  each  class 

was  first  obtained  and  the  average  coefficient  for  all  classes 

was  calculated  therefrom.     In  Table  XVIII  the  coefficients  of 

reliability  thus  obtained  are  presented. 


TABL3 


CORRELATION  BET\VEEN  THE  IvIEAI^J  SCORES,  BY  TRAITS, 
i^CEIYED  BY  17  INSTRUCTORS  FROM  429  STUDENTS 
ON  RATINGS  LiADE  TWO  YIEEXS  AE'ART. 


Number  of 

Classes 

Students 

r 

1 

35 

.7988 

.041 

2 

20 

,9389 

.017 

3 

13 

.8473 

.052 

4 

17 

.5539 

.114 

5 

17 

.5915 

.106 

6 

23 

.7267 

.017 

7 

31 

.6703 

.066 

8 

35 

.9284 

.017 

9 

41 

.9774 

.004 

10 

35 

.9024 

.021 

11 

21 

.8929 

.028 

12 

16 

.9026 

.032 

13 

30 

.8976 

.023 

14 

27 

.9074 

.024 

15 

34 

.7069 

.059 

16 

17 

.8865 

.037 

17 

17 

.9162 

.028 

Average 

25,2 

.8625 

.008 

Table  XVIII  should 

be  read 

as  follows: 

There  were 

17  classes 

involved  in  this 

study. 

The  number  of 

students  in 

\ 

\ 
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the  first  class  was  35.    The  coefficient  of  correlation  between 
the  mean  scores  on  each  trait,  obtained  at  two  ratings  made  two 
weeks  apart  is  .7988.     If  the  mean  scores  obtained  from  a  num- 
ber of  classes,  of  the  same  size  and  composition,  were  correlat- 
ed, 50  per  cent  of  the  coefficients  of  correlation  would  lie 
between  .7988  ±  .041  or  between  .8398  and  .7578.    It  is  practic- 
ally certain  that  the  true  coefficient  of  correlation  lies  be- 
tween .7988  ±  4  X  .041,  or  between  .9628  and  .6348.     The  data 
for  the  other  classes  are  read  in  the  same  manner. 

Except  for  classes  4,  5,  and  7,  the  obtained  coeffi- 
cients of  correlation  are  high  enough  and  the  probable  errors 

7 

small  enough  to  indicate  a  rather  high  degree  of  reliability. 
The  average  coefficient  of  correlation  of  .8635  ±  .008  is  based 
upon  the  scores  received  from  the  429  students. 

Another  comparison  of  the  same  mean  scores,  which  were 
correlated  as  shown  in  Table  XVIII,  was  made  by  subtracting  the 
mean  scores  received  on  each  trait,  on  the  two  ratings  two  weeks 
apart,  and  by  averaging  the  differences  thus  obtained  on  all 
the  17  traits.     The  data  secured  in  this  way  are  presented  in 
Table  XIX. 

"^Ibid.  pp.  170-71. 

H.  A.  Wallace  and  George  W.  Snedecor,  Correlation  and  Machine 
Calculation.    Ames,  Iowa:    Iowa  State  College  of  Agriculture 
and  Mechanic  Arts,  Official  Publication,  Vol.  XXX.  No.  4. 
1931.     pp.  62-64. 

Harold  0.  Rugg,  Statistical  Methods  Applied  to  Education, 
Boston:  Houghton  Mifflin  Co.    1917.    p.  256. 
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TABLE  XIX 

DIFFER3NCES  BETVffiEN  mm  SCORES  OBTAI^TED  ON  RATINGS 
ILIADE  TWO  ^VEEPCS  APART. 


Class 

Number  of 
Students 

Mean  difference 
between  mean 
scores  per  trait 

SD 
of 

of  dist. 
the  mean 
scores 

 WW 

■m 

1 

35 

1.76 

1.78 

.  20 

2 

20 

T  CO 

1.58 

.82 

.12 

3 

13 

1 . 58 

1.07 

.20 

A 

4 

17 

1.47 

1.41 

.  23 

5 

17 

2. 58 

1.73 

.28 

6 

23 

3.70 

O  O 

2.88 

,  40 

7 

31 

3.  35 

2.61 

.32 

8 

35 

1.58 

1.85 

.21 

y 

41 

1.17 

.00 

Art 

•  09 

10 

35 

1.29 

1.25 

.14 

11 

21 

2,29 

2.11 

.31 

Ic 

xo 

13 

30 

3.82 

3.01 

.55 

14 

27 

0.94 

1.05 

.14 

15 

34 

2.41 

2.25 

.26 

16 

17 

.76 

.09 

.01 

17 

17 

2.11 

2.17 

.35 

Averages 

25.2 

2.01 

1.73 

.25 

Table  XIX  should  be  read  as  follows:     In  the  first 


class,  35  students  rated  their  instructor  on  two  occasions  two 
weeks  apart.    The  average  of  the  differences  between  the  mean 
scores  obtained  by  these  two  ratings  on  each  trait  is  1.76. 
Sixty-eight  per  cent  of  the  17  differences  thus  obtained  lie 
between  1.76  ±  1,78,  or  between  0  and  3.54.     If  corresponding 
data  were  derived  from  other  classes  of  like  size  and  composition 
the  chances  are  even  that  the  mean  difference  would  fall  between 
1.76  ±  .20  or  between  1.96  and  1.56. 

The  fact  that  the  standard  deviation  is  larger  than 
the  average  of  the  obtained  differences,  indicates  that  the 
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distribution  of  these  differences  was  considerably  skewed.  This 
is  caused  by  the  influence  of  a  relatively  few  very  large  dif- 
f erences. 

The  differences  between  the  mean  scores  obtained  from 
two  ratings  made  two  weeks  apart  are  shown  in  Table  XIX  to  be 
comparatively  small,  when  the  possible  range,  0-100,  is  consid- 
ered.   The  standard  deviations  are  small  enough  to  indicate 
marked  concentration  of  the  obtained  differences  about  the  mean 
for  each  of  the  different  traits. 

b.  Correlation  between  the  Sum  of  the  Scores  on  All 
the  Traits,  Given  by  Each  of  the  429  Students, 
to  his  Instructor,  on  Two  Ratings  Made  Two  Y/eeks 
Apart , 

In  the  computation  of  this  correlation  the  same  raw 
data  were  used  as  in  the  preceding  comparison,  but  they  were 
thrown  together  in  a  different  manner.     In  this  comparison 
the  sum  of  the  scores  given  by  each  individual  student,  on 
all  the  17  items  of  the  scale,  on  the  occasion  of  the  first 
rating,  is  correlated  with  the  sum  of  the  scores  given  by  the 
same  student  two  weeks  later.     The  averages  might  just  as  well 
have  been  used,  but  since  in  calculating  this  particular  cor- 
relation coded  values  were  used,  whicn  yielded  small  numerical 
quantities  for  the  sums,  there  was  no  particular  advantage  in 
using  means. 

In  Table  XX  the  coefficients  of  reliability  thus 
obtained  are  presented. 

Q 

Henry  E.  Garrett,  Op.   cit.  p. 86^ 
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TABLE  XX 

CORRELATION  BET^VEEN  THE  TOTAL  SCORES,  ON  ALL  17 
TRAITS,  GIVEN  BY  EACH  OF  429  STUDENTS 
IN  17  CLASSES,  ON  TWO  RATINGS  TWO 
V/EEKS  APART. 


Number  of 

Class 

Students 

r 

PEr 

1 

35 

.6172 

.071 

2 

20 

.9065 

.028 

3 

13 

.6679 

.117 

4 

17 

.6856 

.088 

5 

17 

•  7260 

.076 

6 

23 

.8599 

.036 

7 

31 

.7350 

.055 

8 

35 

.6261 

.070 

9 

41 

.7557 

.046 

10 

35 

.5839 

.075 

11 

21 

.8556 

.040 

1]  1 

•  JL  J.  .i. 

13 

30 

.9035 

.023 

14 

27 

15 

34 

.7264 

.055 

16 

17 

.6519 

17 

17 

.8593 

.048 

Average 

S5.2 

.7592 

.014 

Table  XX 

should  be 

read 

as  follows: 

In 

the  first 

class  containing  35  students, 

the 

coefficient 

of  1 

correlation 

between  the  first 

and  second 

ratings  made  by  each 

individual 

student  was  ,6172. 

If  coefficients  v/ere  obtained 

for  a  number 

of  classes  of  like  size  and  composition,  the  chances  are  50  in 
100,  that  such  coefficients  v/ould  lie  between  .617  t  .071  or 
between  .688  and  .546.     It  is  practically  certain  that  the  true 
coefficient  of  correlation  will  lie  within  the  liciits  .617  ± 


4  X  .071  or  between  90.1  and  33.3. 

In  interpreting  the  data  in  Table  XX  it  must  be  remem- 
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bered  that  they  refer  to  the  correlation  between  two  ratings 
made  by  the  individual  student,  and  not,  as  in  Table  XVIII  to 
correlations  between  the  means  of  class  scores.     This  accounts, 
of  course,  for  the  reduced  values  of  the  coefficients.  liow- 
ever,  the  coefficients  are  still  large  enough  to  indicate  a 
rather  high  degree  of  consistency^  in  the  judgment  of  the 
College  Teachers  Rating  Scale, 

Because  the  judgment  of  any  one  individual  student 
would  very  rarely,  if  ever,  be  used  to  indicate  the  teaching 
ability  and  personal  fitness  of  his  instructor,  the  practical 
value  of  the  coefficients,  presented  in  Table       may  be  ques- 
tioned,   Hov/ever,  since  the  consistency  or  reliability  of 
student  judgraent  is  the  main  problem  of  this  phase  of  the  in- 
vestigation, it  was  considered  pertinent  to  present  any  evidence 
of  such  consistency  as  the  investigation  disclosed.     The  numer- 
ical values  of  the  reliability,  coefficients  submitted  in  Table 
XX  suggest  that  the  judgment  of  students  concerning  the  teach- 
ing ability  of  their  instructors  possesses  a  greater  degree  of 
reliability  than  is  oftentimes  accorded  to  it, 

A  still  more  severe  test  of  the  reliability  of  stu- 
dent judgraent  was  made  by  correlating  the  tuo  scores,  on  each 
individual  trait,  given  to  his  instructor  by  each  student,  in 
ratings  made  two  weeks  apart. 

Four  hundred    twenty-nine  pairs  of    ratings  were 


Ibid,  p,  269. 

H.  A.  7/allace  and  George  kV.  Snedecor,     Op.  cit,  pp,  68-64, 
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used  in  calculating  the  data  presented  in  Table  XKI  and  XXII 
which  follow.    Correlations  were  run  separately  for  each  trait 
in  order  to  make  possible  a  comparison  between  them. 

TABLE  XXI 


CORRELATION  BETVffiEN  THE  TWO  SEPARATE  MD  IITOIVIDUAL 
RATINGS,  ON  EACH  TRAIT,  GIVEN  TY/0  VilEEKS 
APART  BY  EACH  OF  429  STUDENTS 


Traits 

numbered  as  on  scale 

r 

PEp 

I. 

Preparation  for  class  meetings 

.6638 

.018 

II. 

Interest  and  enthusiasm  in  subject 

.o840 

.022 

III. 

Ability  to  arouse  interest  in  student 

.6070 

.021 

IV. 

Organization  of  course 

.6697 

.018 

V. 

Scholarship 

.6727 

.017 

VI. 

Ability  to  express  thought 

.6340 

.020 

VII. 

Enunciation 

.6716 

.017 

VIII. 

Thinking  demanded  of  students 

.6254 

.020 

IX. 

Assignments 

.7087 

.016 

X. 

Leading  discussions  and  questioning 

.7020 

.016 

XI. 

Sense  of  proportion 

.7252 

.015 

XII. 

Feeling  between  instructor  and  students. 8065 

.011 

XIII. 

Sense  of  humor 

.5579 

.022 

XIV. 

Self-confidence 

.6017 

.014 

XV. 

Tolerance  and  liberality 

.7393 

.021 

XVI. 

Personal  appearance 

.5542 

.022 

XVII. 

Personal  peculiarities 

.6736 

.018 

Average 

.6707 

.018 

Table  XXI  should  be  read  as  follows:     The  coefficient 


of  correlation  between  the  two  ratings  of  429  students  on  Trait 
I  is  ,6638.     If  corresponding  coefficients  were  obtained  for  a 
number  of  groups  of  classes  of  the  like  size  and  from  the  same 
population,  the  chances  are  50  in  100  that  the  coefficients 
would  fall  between  .6638  t.  .018  or  between.  . 6818  and  .6458.  It 
is  practically  certain  that  the  true  coefficient  of  correlation 
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will  lie  within  the  limits  .6638  j:  4  x  .018  or  between  .7358 
and .5918. 

The  coefficients  of  reliability  presented  in  Table 

XXI,  while  not  so  large  as  those  reported  in  the  Tables  XVIII 

10 

and  XX,  are  large  enough  to  be  significant.    The  probable  errors 

of  the  coefficients  are  small  enough  to  indicate  a  high  degree 

11 

of  reliability  for  the  coefficients.      Furthermore,  an  examina- 
tion of  the  raw  data  convinces  one  of  the  failure  of  the  coef- 
ficient of  correlation  to  give  an  adequate  measure  of  the  actual 
correspondence  or  similarity  existing  between  the  two  sets  of 
variables  being  compared.    This  weakness  of  the  correlation 
method  under  certain  conditions  has  been  discussed  fully  in 
Chapter  IV. 

Another  measure  of  the  amount  of  correspondence  be- 
tween the  two  ratings,  being  compared,  supplementary  to  the  cor- 
relation method,  was  sought  by  computing  the  difference  between 
them.     The  obtained  differences  are  presented  in  Table  XXII. 

Table  XXII  should  be  read  as  follows:     The  mean  of 
the  429  differences  obtained  by  subtracting  the  individual  scores 
obtained  in  the  two  ratings  on  Trait  I,  and  by  averaging  the 
differences  thus  obtained,  was  4.86.     Sixty-eight  per  cent  of  the 
differences  between  each  pair  of  individual  scores  on  Trait  I 
fall  within  4.86  ±  6.32  or  between  0  and  11,18.    Practically  all 
(99  in  100)  of  the  differences  between  each  pair  of  scores  fall 

1°  Ibid,  ip.  62-64. 

Henry  E.  Garrett.     Op.   cit.  p. 170, 
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wlthin  4.86  ±  3  x  6.32  or  between  0.00  and  23.82.     If  corre- 
sponding data  were  derived  from  other  groups  of  classes  of  like 
size  and  composition  the  chances  are  even  that  the  mean  of  the 
differences  would  lie  between  4.86  ±  .20  or  between  5.06  and 
4.66.    We  may  be  practically  certain  that  the  true  mean  lies 
between  4.86  ±.4  x  ,20  or  between  5.66  and  4.06.     The  data  for 
each  of  the  other  traits  are  read  in  the  saiae  manner,  using 
their  corresponding  values, 

TABL3  XXII 


DIFFERENCE  BETV/EEN  THE  TWO  SEPARATE  Al^ID  INDIVIDUAL 
RATI2TGS,  ON  EACH  TRAIT,  GIVEN  TWO  YfflEKS 
APART  BY  SACH  OF  429  STUDENTS 


Traits 

Mean  of 
Differences 

SD  of  distribution 
of  differences 

PE 

m 

I 

4.86 

6.32 

.20 

II 

4.68 

6.34 

.20 

III 

8.33 

10.73 

.35 

IV 

7.49 

8.27 

.27 

V 

4.64 

5.50 

.18 

VI 

7.24 

8.52 

.28 

VII 

6.85 

7.94 

.26 

VIII 

6.70 

8.61 

.28 

IX 

8.40 

9.32 

,30 

X 

7.32 

7.98 

.26 

XI 

6.57 

7.42 

.24 

XII 

6.65 

8.33 

.27 

XIII 

6.78 

8,28 

.27 

XIV 

5.67 

7.35 

.24 

XV 

5.99 

7.53 

.24 

XVI 

4.10 

5.99 

.19 

XVII 

5.99 

6.48 

.21 

Average 

6 . 36 

7.70 

.25 

In  obtaining  the  data  in  Table  XXII  the  difference  be 


tween  the  two  scores  given  by  each  student  was  first  calculated. 
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The  standard  deviations  submitted  in  Table  XXII  refer  to  the 
distribution  of  the  means  of  the  differences  between  the  two 
ratings  given  by  each  of  the  429  students. 

It  should  be  remembered,  when  interpreting  the  data 
presented  in  Table  XXII,  that  the  total  range  of  the  scale  is 
100,  making  it  possible  to  translate  the  obtained  differences 
directly  as  percentages  of  the  possible  difference.     The  obtain- 
ed mean  differences  of  from  4  to  8  per  cent  of  the  possible 
difference,  may  properly  be  regarded  as  indicative  of  a  high 
degree  of  correspondence  between  the  two  ratings. 

c.  Comparison  of  the  Mean  Scores  Received  by  a  Large 
Number  of  Instructors  from  Classes  Taught  During 
the  Sane  Terms. 

It  has  already  been  pointed  out  that  the  coefficient 
of  correlation,  under  certain  conditions,  fails  to  measure 
adequately  the  correspondence  between  two  sets  of  variables. 
An  inspection  of  the  raw  data  led  the  investigator  to  believe 
that  the  correspondence  or  similarity  between  the  scores  ob- 
tained from  two  separate  ratings  was  greater  than  the  numerical 
values  of  the  coefficients  of  correlation  between  them  indicat- 
ed, and  the  difference  between  the  two  scores  was  thought  to 
indicate,  more  adequately,  the  correspondence  between  them. 

In  determining  these  differences  only  the  means  of 
the  scores  given  by  various  classes  were  used.     Two  groups  of 
classes  were  studied.     The  first  group  was  made  up  of  190 
pairs  of  classes.    The  two  classes  in  each  pair  were  in  the 
same  course  and  taught  by  the  same  instructor  in  the  same  tena. 
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These  two  groups  were  first  handled  separately  as  shown  in 
Table  XXIII  and  then  compared  v/ith  one  another  in  Table  XXIV. 
The  mean  scores  of  the  two  classes  in  each  pair  was  subtracted 
one  from  the  other  and  the  means  of  the  differences  thus  ob- 
tained with  their  respective  standard  deviations  were  calculat- 
ed. 


TABLE  XXIII 


DIFFERENCES 

BET\VEEN 

MEAN  SCORES  GIVEN 

TO  SAME 

INSTRUCTORS  BY 

CLASSES 

TAUGHT  IN 

SAJ.IE 

TERM 

:Between  Two 

Classes 

in  the 

: Between  Two  Classes 

in  dif- 

Traits 

:Same  Subjec 

t  {190  pairs) 

:ferent  Subjects  (341  pairs) 

:    Mean  ; 

Sadist- 

PE 

:  Mean 

'  SD^ist- 

: difference : 

• 
• 

: difference 

•  • 

•  • 

I 

O  •  \Jei 

5.67 

.28 

:  5.67 

5.95 

.22 

II 

3.51 

3.34 

.16 

4.40 

4.42 

.16 

III 

5.90 

5.65 

.28  : 

8.48 

7.65 

.28 

IV 

5.61 

6.03 

.29 

:  7.23 

6.40 

.23 

V 

4.04 

4.41 

.21 

:  5.02 

5.00 

.19 

VI 

4.82 

4.91 

.24 

:  6.14 

5.88 

.21 

VII 

4.33 

4.46 

.22  . 

:  4.82 

6.01 

.22 

VIII 

5.53 

5.02 

.24 

6.64 

5.92 

.22 

IX 

5.23 

4.79 

.24 

:  6.58 

6.35 

.24 

X 

5.71 

5,49 

.26  ■ 

:  7.93 

7.01 

.26 

XI 

5.09 

4.07 

.19 

:  6.51 

5.52 

.20 

XII 

6.16 

7.03 

.34 

:  7.78 

8.84 

.34 

XIII 

B.91 

6.55 

.32  . 

:  8.20 

7.85 

.28 

XIV 

4.49 

4,17 

.20  : 

:  5.60 

5.25 

.19 

XV 

5.25 

5.65 

,28  : 

:  6.26 

5.93 

.29 

XVI 

3.19 

4.24 

.21  ' 

4.03 

3.90 

.14 

XVII 

4.27 

4.33 

•  S  S 

5.87 

7.05 

.26 

Average 

5.00 

5.05 

.25  : 

6.30 

6.23 

.23 

Table  XXIII  read  as  follows:     The  mean  of  the  dif- 
ferences between  the  ratings  given  by  Trait  I  by  the  two 
classes  in  each  pair  of  classes  in  the  same  course  taught  by 
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the  same  Instructor  in  the  same  term  is  5.02.    Sixty-eight  per 
cent  of  these  differences  fall  within  5,02  ±  5.67  or  between 
0.00  and  10.69.     Furthermore,  the  chances  are  50  in  100  that 
the  obtained  mean  difference  of  5.02  does  not  diverge  from  the 
true  mean  of  the  differences  by  more  than  ±  ,28,     It  is  practic- 
ally certain  that  the  true  mean  difference  l^es  within  the  limits 
5.02t4  X  .28  or  between  3.90  and  6.14. 

This  interpretation  of  Table  XXIII  shows  the  means  of 
the  differences  between  the  two  sets  of  scores  to  be  highly  re- 
liable.    The  comparatively  large  standard  deviations  from  the 
mean  difference,  however,  indicates  a  wide  range  or  spread  in 
the  distribution  of  the  difference  .     The  fact  that  for  some 
traits  the  standard  deviations  are  greater  than  the  means  in- 
dicates a  badly  skewed  distribution  of  the  differences  and  an 
inspection  of  the  individual  differences  confirms  this.     In  the 
case  of  some  of  the  pairs  of  classes  compared,  the  difference 
between  the  mean  scores  was  several  times  greater  than  the  mean 
difference  of  the  scores  of  all  the  pairs  in  all  the  classes. 
These  large  differences,  though  not  at  all  numerous,  tended  to 
increase  considerably  the  mean  of  the  differences  and  also  the 
standard  deviations  of  the  differences  from  their  mean.  The 
occurrence  of  these  large  differences  demonstrates  that  oc- 
casionally the  judgment  of  stuaents  concerning  an  individual  in- 
structor, even  when  the  means  of  the  class  scores  are  compared, 
is  very  inconsistent  and  unreliable.    Hov/ever,  the  mean  dif- 
ferences in  the  case  of  all  the  traits  must  be  regarded  as 
indicating  a  close  correspondence  between  the  two  sets  of 
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scores,  7/hen  the  possible  range  of  the  means  is  froiu  0  to  100. 
It  must  also  be  remembered  that  the  scores  which  are  subtract- 
ed to  obtain  the  differences  are  given  in  each  case  by  two 
different  classes,  and  that,   even  if  the  scale  were  absolutely 
reliable,  the  scores  given  by  different  classes  irii,5ht  differ 
to,  at  least,  an  appreciable  extent.    The  means  of  the  differ- 
ences between  the  scores  made  by  the  saiae  classes  on  two  rat- 
ings tv/o  weeks  apart,    (see  Table  XIX)  are  seen  to  be  consider- 
abl    smaller  than  those  show  in  Table  XXIII,  which  are  differ- 
ences between  the  means  from  different  classes.     It  might  well 
be  reasoned  that  the  teaching  effectiveness  of  any  given  in- 
structor would  vary  less  within  a  period  of  two  weeks  in  the 
same  class,   than  it  would  betv/een  two  different  classes.  If 
such  reasoning  is  sound,  the  smaller  differences  obtained 
between  two  ratings  made  in  the  same  class  offer  added  evidence 
of  the  reliability  of  the  scale,  since  the  students,  through 
its  use,  are  able  to  observe  and  record  this  difference. 

Somewhat  the  same  line  of  reasoning  may  be  applied 
to  the  differences  found  between  the  means,  when  pairs  made  up 
of  classes  in  different  courses  are  compared  with  those  with 
classes  in  the  same  course.     The  statistical  significance  of 
these  differences  is  presented  in  Table  XXIY. 

Table  XXIV  is  read  is  follows:     The  mean  of  the  dif- 
ferences between  the  scores  given  by  341  pairs  of  classes  in 
different  courses  is  greater  by  .65  than  the  mean  of  the  differ- 
ences between  scores  given  by  190  pairs  of  classes  in  the  same 
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courses.     The  chances  are  50  in  100  that  the  obtained  differ- 
ence of  ,65  does  not  diverge  from  the  true  difference  by  more 
than  ±  .35.    The  chances  are  89  in  100  that  the  true  differ- 
ence between  the  two  means  in  question  is  greater  than  zero 
The  data  for  the  other  traits  are  read  in  the  sane  way. 

TABLE  XXIV 

SIGNIFICANCE  OF  DIFFERENCES  BETVfflEN  RATINGS  I^DE 
BY  CLASSES  IN  SAME  COURSES  AND  BY  CLASSES 
IN  DIFF2RENT  COURSES.    (SEE  TABUS  XXIII) 
(190  pairs  in  same  courses;  341  pairs  in  different  courses.) 


Traits 

Diff, 

P^diff . 

Diff. 
P^diff . 

Chances  in  100 

I 

.65 

.35 

1.86 

89 

II 

.89 

.23 

3.87 

99 

III 

2.58 

.39 

6.61 

100 

IV 

1.62 

.36 

4.50 

100 

V 

.98 

.28 

3.50 

99 

VI 

1.32 

.31 

4.26 

100 

VII 

,49 

.30 

1.63 

86 

VIII 

1,11 

.32 

3.47 

99 

IX 

1.35 

.34 

3.97 

100 

X 

2,22 

.36 

6.17 

100 

XI 

1.42 

.28 

5.07 

100 

XII 

1.62 

.47 

3,45 

99 

XIII 

1,29 

.42 

3.07 

98 

XIV 

1.11 

•  28 

3.96 

100 

XV 

1.03 

.34 

3.03 

98 

XVI 

.84 

.25 

3.36 

99 

XVII 

1,60 

.33 

4,85 

100 

Average 

1.30 

.34 

3.82 

There  is  evidently  not  a  great  deal  of  difference  ob- 


served by  college  students  in  effectiveness  with  which  an  in- 
structor teaches  classes  in  different  courses.     The  differences 
between  the  means  of  the  differences  between  the  ratings  given 
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by  classes  in  the  same  course  and  the  means  of  the  differences 
between  the  ratings  given  by  classes  in  different  courses  is 
shown  in  Table  XXIV  to  be  small,  averaging  1,30,  with  small 
deviations  from  this  average.    However,  the  chances  are  high, 
in  all  the  traits,   that  the  differences  though  small,  are 
true  differences,"^^  i.e.,  differences  greater  than  zero.  The 
small  numerical  value  of  the  standard  deviations  of  the  differ- 
ences for  the  various  traits  indicates  that  these  differences, 
while  real,  will  not  often  be  large  enough  to  be  very  signifi- 
cant, when  comparing  the  ratings  given  to  instructors  by  classes 
in  different  courses.     This  point  is  made  use  of  in  the  study 
of  the  influence  of  class  size  upon  the  ratings  given  to  in- 
structors,  (see  Chapter  X),  where  it  v/as  necessary,  in  order 
to  increase  the  nuraber  of  observations,   to  compare  the  ratings 
given  by  classes  in  different  courses. 


Henry  E.  Garrett,  Op.   cit.  p. 135. 
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CHAPTSH  VIII 
EVIDENCES  OF  VALIDITY 

The  determination  of  the  validity  of  the  College 
Teachers  Rating  Scale  is  the  central  problem  of  this  chapter. 

The  validity  of  any  test  or  scale  refers  to  the  ex- 
tent to  which  it  measures  that  which  it  purports  to  measure.''' 
Two  interpretations  have  been  applied  to  the  use  of  the  term 
"validity"  in  connection  with  rating  scales,  and  these  two 
interpretations  depend  upon  that  which  the  scale  is  supposed 
to  measure.     If  it  is  held  that  what  the  scale  measures  is 
merely  the  students*  judgment  of  the  instructor,  then  the  re- 
liability of  the  scale  is  also  a  measure  of  its  validity,  that 
is,  the  two  terms  are  synonymous  in  meaning.    This  is  the  in- 
terpretation given  to  the  term  by  Reramers  and  Brandenburg.^ 
Based  upon  this  interpretation  of  the  term,  the  validity  of 
the  College  Teachers  Rating  Scale,  developed  in  this  study,  is 
well  established  by  the  evidence  presented  in  Chapter  VII,  If, 
on  the  other  hand,  it  is  held  that  the  scale  measures  the 
quality  of  instruction  the  term  "validity"  would  not  be  syn- 
onymous with  "reliability",  but  v/ould  refer  to  the  extent  to 
which  student  judgment  exercised  through  the  medium  of  the  scale 

"^Karl  J,  Holzinger,  Statistical  IJethods  for  Students  in  Education 
Boston:     Ginn  &  Co.     1928  p.  168. 

2 

H.  H.  Remmers,  and  G.  E.  Brandenburg,  "Experimental  Data  on 
the  Purdue  Rating  Scale  for  Instructors",  Educational  Admin- 
instratlon  and  Supervision,  XIII  (November  1927 ) ,pp . 519-527 . 
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and  recorded  upon  it,  measured  the  teaching  ability  and  personal 
fitness  of  the  instructor. 

The  latter  interpretation  of  "validity"  is  the  one  adopt- 
ed in  this  investigation.    A  large  share  of  the  energy  and  ex- 
pense expended  in  this  study  was  expended  on  the  attempt  to  ob- 
tain a  measure  of  the  validity  of  the  College  Teachers  Rating 
Scale. 

The  technique  utilized  in  the  determination  of  the 
validity  of  the  scale,  briefly  stated,  consisted  of  three  main 
parts:     (1)  the  careful  recording  of  observations,  made  by  the 
investigator,  in  four  consecutive  meetings  of  the  classes  of  each 
of  27  instructors:    (2)  the  evaluation  or  grading  of  these  obser- 
vations by  three  recognized  authorities  on  method,  working  inde- 
pendently of  one  another;    (3)  comparison  of  the  mean  scores  given 
to  each  instructor  by  the  three  judges,  with  those  given  to  the 
same  instructor  by  the  students  in  the  observed  classes.     The  en- 
tire technique  has  been  described  in  detail  in  Chapter  IV. 

In  setting  forth  the  findings  of  this  part  of  the  study 
the  v/riter  decided  to  present  all  the  data  in  considerable  detail, 
in  order  that  those  interested  in  evaluating  the  soundness  of  the 
method,  which  is  not  a  usual  one  for  the  purpose,  could  give  the 
obtained  raw  data  careful  scrutiny. 

Because  of  the  v/ide  variation  in  many  of  the  ratings 
given  by  the  three  judges  and  because  their  small  number  prevents 
the  use  of  the  standard  statistical  measures  of  variability,""'  all 
the  ratings  of  all  the  judges  for  each  instructor  are  shown  in  the 
•^arl  J.  Holzinger,  Op.  cit.pp.3-4,  79i 
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Table  XXV  which  follows. 


TABLE  XXV 


SCORES  GIVEN  TO  EACH  OF  27  INSTRUCTORS,  BY  THREE 
JUDGES,  ON  THE  BASIS  OF  RECORDED  OBSERVATIONS 
OF  HIS  INSTRUCTION 


CO 
•H 

cd 

E-i 


u 
o 
■p 
o 

+> 

CO 

a 


Judges 


A 


B 


o  to  w 

CD  0) 

(d  ^  o 
(D  0  a 
S  K>  c/1 


o 
-p 
o 

03 
J=L 


A 


Judges 


B 


O  W  CO 

CD  (D 

d o 

<D  o 


1  1 

60 

86 

95 

80.33 

2  60 

94 

97 

83.66 

II 

81 

96 

93 

90.00 

70 

95 

94 

86.53 

III 

70 

90 

87 

82.33 

60 

98 

93 

83,66 

IV 

70 

76 

88 

78.00 

50 

97 

92 

79.66 

VI 

66 

90 

76 

77.33 

90 

98 

90 

92.66 

VIII 

50 

89 

80 

73.00 

80 

99 

88 

89.00 

IX 

50 

70 

70 

63.33 

50 

85 

90 

75.00 

X 

66 

80 

87 

76.66 

80 

97 

88 

88.33 

XI 

60 

86 

88 

78.00 

70 

90 

88 

82.56 

I  3 

60 

70 

90 

73.33 

4  60 

76 

60 

65,33 

II 

80 

90 

85.00 

55 

66 

67 

62.66 

III 

70 

70 

90 

76.66 

40 

41 

60 

47.00 

IV 

60 

61 

80 

67.00 

30 

70 

40 

46.66 

VI 

70 

81 

57 

69.33 

80 

80 

70 

76.66 

VIII 

60 

70 

63 

64.33 

40 

50 

45.00 

IX 

30 

50 

40 

40.00 

40 

40 

65 

48.33 

X 

60 

70 

57 

62.33 

30 

21 

50 

33.66 

XI 

60 

70 

60 

63,33 

50 

80 

70 

66.66 

I  5 

70 

82 

70 

74.00 

6  50 

51 

36 

45.66 

II 

80 

83 

89 

83.66 

50 

40 

50 

46.66 

III 

56 

66 

70 

64.00 

30 

41 

20 

30.33 

IV 

60 

80 

80. 

73.33 

40 

30 

30 

33.33 

VI 

90 

91 

70 

83.66 

90 

50 

66 

68.66 

VIII 

65 

60 

50 

58.33 

30 

21 

12 

21.00 

IX 

50 

41 

20 

33.66 

20 

30 

10 

20.00 

X 

70 

60 

60 

63.33 

16 

20 

20 

18.56 

XI 

50 

70 

80 

66.66 

50 

30 

18 

35.56 
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TABLS  XXV  (continued) 


m     I  Judges        ^  ;     ;  Judges 

  «  M  


I      7     96  94  85  91.56:  :     8      96  90  96  94.00 

II  96  93  78  89.00:  :  96  100  96  97.33 

III  90  82  60  77.33:  :  90  90  96  92,00 

IV  95  91  60  82.00:  :  95  99  95  96.33 

VI  95  93  85  91.00:  :  95  95  93  95.66 

VIII  95  86  67  82.66:  :  90  99  93  94.00 

IX  96  90  52  79.33:  :  95  89  80  88.00 

X  96  90  50  78.66:  :  90  90  93  91.00 

XI  96  93  53  80.66:  :  90  94  92.00 


1     9       95  97  80  90.66:  :  10      60  90  73  74.33 

II  90  100  77  89.00:  :  70  80  70  73.33 

III  95  99  63  85.66:  :  40  69  50  53.00 

IV  95  98  90  94.33:  :  40  80  60  60.00 

VT  96  99  78  91/00:  :  90  80  77  82.33 

VIII  95  100  79  91.33:  :  40  81  32  51.00 

IX  96  97  75  89.33:  :  40  51  20  37.77 

X  96  100  67  87.56:  :  40  59  50  53.00 

XI  90  99  88  92.33:  :  50  70  50  56.56 


I    11    85  80  83  82.66:  :     12    90  96  89  91.66 

II  90  80  82  84.00:  :  90  91  87  91.66 

III  86  70  70  75.33:  :  90  98  80  89.33 

IV  90  79  80  83.00:  :  86  97  90  89.00 

VI  81  80  80  80.33:  :  90  97  88  91.66 

VIII  95  90  70  85.00:  :  80  98  86  88.00 

IX  90  71  90  83.66:  :  80  75  61  68.55 

X  95  79  70  81.33:  :  90  95  84  89.66 

XI  95  78  89  87.33:  :  90  90  83  87.66 


I     13     60  90  88  79.33:  :  14      80  76  86  81.00 

II  90  89  88  89.00:  :  85  86  85  85.33 

III  40  80  70  63.33:  :  80  77  80  79.00 

IV  50  79  80  69.66:  :  70  77  84  77.00 

VI  90  97  88  91.66:  :  80  80  82  80.66 

VIII  35  77  76  62.66:  :  57  88  75  73.00 

IX  30  67  77  58.00:  :  34  60  47.00 

X  26  76  78  60.00:  :  60  60  68  62.66 

XI  50  83  66.50:  :  70  60  80  70.00 
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TABLE  XXV  (continued) 


M 

1 

Judges 

Mean  of 
Judge  s 
Scores 

:  1 

Judg 

es 

C-4 

Trail 

Instru( 

tO] 

A 

B 

C 

Instru( 

tO] 

A 

B 

C 

Mean  o: 
Judges 
Scores 

I 

15 

fU 

88.66 

:  16 

oO 

no 

90 

68.33 

II 

97 

Q7 

94.66 

75.33 

III 

80 

95 

62 

79.00 

50 

50 

50 

50.00 

IV 

70 

96 

93 

76.33 

50 

41 

64 

51.66 

VI 

80 

99 

88 

89.00 

86 

67 

78 

77.00 

'\TT  TT 

50 

98 

75 

/  4:.  OO 

40 

41 

65 

4kO.  OO 

IX 

60 

96 

74 

76. 66 

50 

30 

31 

37 . 00 

X 

65 

88 

65 

72.66 

46 

30 

58 

44.66 

XI 

70 

94 

60 

74.65; 

60 

51 

68 

59.66 

I 

17 

75 

90 

97 

87 , 33; 

,  18 

oO 

96 

97 

92 ,66 

II 

85 

90 

96 

90. 33; 

90 

95 

97 

94.  00 

III 

90 

90 

83 

87 . 66 : 

90 

100 

97 

95 . 66 

IV 

90 

90 

90 

90 . 00 : 

90 

96 

96 

94.  00 

VI 

86 

89 

80 

85.00: 

95 

93 

95 

93.  66 

VIII 

90 

90 

94 

91,33; 

96 

100 

96 

97.33 

IX 

40 

70 

55 

55.00: 

68 

97 

96 

87.00 

X 

89 

90 

91 

90.00: 

96 

98 

96 

96.66 

XI 

80 

81 

81 

80.66: 

90 

96 

97 

94.33 

I 

19 

70 

91 

96 

85.66: 

20 

ex.  C\ 

1  D 

/  0 

70. 33 

II 

86 

90 

97 

91.00: 

60 

80 

74 

71 . 33 

III 

80 

90 

92 

87 .33: 

70 

86 

75 

77  .00 

IV 

80 

90 

90 

86.66: 

60 

87 

50 

65.66 

VI 

95 

89 

90 

91.66: 

60 

61 

20 

47  ,00 

VIII 

90 

100 

93 

94.33: 

50 

69 

30 

49.66 

IX 

65 

79 

60 

68.00: 

50 

41 

20 

37.00 

X 

80 

100 

90 

90.00: 

60 

39 

10 

36.33 

XI 

85 

90 

86 

87.00: 

60 

71 

36 

55.66 

I 

21 

86 

81 

92 

86.33: 

22 

n  K 
YO 

oU 

y  u 

81.66 

II 

90 

89 

90 

89.66: 

90 

90 

90 

90.00 

III 

90 

90 

91 

90.33: 

90 

89 

90 

89.66 

IV 

85 

80 

90 

85.00: 

77 

89 

89 

85.00 

VI 

96 

90 

90 

92.00: 

95 

90 

90 

91.56 

VIII 

90 

90 

90 

90.00: 

96 

87 

90 

91.00 

IX 

90 

81 

70 

80.33: 

50 

71 

87 

69.33 

X 

90 

91 

90 

90.33: 

90 

87 

90 

89.00 

XI 

85 

90 

87.50: 

90 

88 

82 

86166 
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TABLE  XSV  (continued) 


(0 

1 

Judge  s 

Mean  of 

Judges 

Scores 

1 

■Tucl  ^e  s 

Trail 

o  u  

!->  u 

CQ 

B 

c 

Instruc 
tor 

A 

"B 

c 

Mean  o 
Judges 
Scores 

I 

23  90 

95 

98 

96.00 

:  24 

60 

80 

88 

76 .00 

II 

95 

95 

97 

95.66 

60 

80 

90 

7  3.33 

III 

94 

99 

98 

97 , 00 

70 

70 

90 

76.66 

IV 

85 

95 

98 

94.00 

50 

80 

80 

70.00 

VI 

90 

96 

90 

92 , 00 

75 

80 

91 

92.00 

VIII 

90 

99 

97 

95.33 

70 

70 

90 

76.65 

IX 

90 

100 

98 

96.0)0 

30 

70 

70 

57.00 

X 

94 

100 

97 

97.00 

70 

70 

90 

77.00 

XI 

81 

98 

96 

91.66 

60 

81 

91 

77.00 

I 

25  95 

91 

89 

92.00 

:26 

96 

89 

90 

91.66 

II 

95 

90 

90 

92.00 

90 

90 

90 

90.00 

III 

95 

90 

90 

92.00: 

95 

91 

90 

92.00 

IV 

90 

91 

90 

90.00: 

95 

90 

80 

98.33 

VI 

95 

91 

90 

92.00: 

95 

90 

90 

91,66 

V  111 

90 

90 

87 

QQ  on  • 
ov . uu : 

95 

90 

90 

yi .  OD 

IX 

80 

81 

90 

84.00: 

94 

90 

80 

88.00 

A 

95 

91 

90 

92.00: 

94 

100 

90 

94.66 

XI 

90 

90 

90.00: 

94 

92 

77 

87.77 

I 

27  96 

96 

87 

93.00: 

II 

90 

96 

82 

89.33: 

III 

93 

96 

80 

89.66 : 

IV 

95 

97 

83 

91.66: 

VI 

95 

99 

82 

92.00: 

VIII 

96 

100 

80 

91.33: 

IX 

90 

98 

78 

88.66: 

X 

95 

99 

78 

90,66: 

ZI 

95 

83 

89.00: 

It 

will 

be  observed 

that  there 

is 

a  considerable  var- 

iation  in  the  ratings  or  scores,  given  by  the  three  judges,  and 
in  the  case  of  some  instructors,  enough,  perhaps,  to  justify 
suspicion  being  cast  upon  the  soundness  of  the  method.     A  more 
careful  study  of  this  variation  may  be  made  by  subtracting  the 
lowest  from  the  highest  rating  and  by  throwing  the  differences 
thus  obtained  into  table  XXVI. 
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TABLE  Xni 


DIPFSRIiiilCES  BETfeOHEN  HIGHEST  AITD  LOVrEST  SCORES  GIVEN  BY  THE 
THREE  JUDGES  ON  EACH  TRAIT,  FOR  EACH  INSTRUCTOR 


1  I 

o 

7i  ' 

Traits 

on  Scale 

._,  - 

^  m 

<*H  <D 

H  W  4->. 

I 

II 

III 

lY 

vr 

VII T 

1  jL  J. 

IX 

X 

XI 

« -H  a 

± : 

OO 

Id 

XO 

24 

49 

ivO  .  O 

Ol 

20 

o8 

47 

8 

19 

4C 

17 

20 

27  ,8 

o: 

3U 

10 

20 

20 

24 

10 

20 

13 

10 

14.1 

lo 

12 

20 

40 

10 

10 

25 

29 

30 

21.  3 

12 

y 

14 

20 

21 

15 

30 

10 

30 

17.9 

o : 

14 

10 

21 

10 

40 

18 

20 

4 

32 

18.9 

7 : 

11 

Ic 

30 

35 

10 

32 

A  A 

44 

A 

46 

43 

28.9 

8: 

6 

4 

6 

4 

6 

9 

15 

3 

>• 

4 

6.3 

9 : 

17 

23 

36 

8 

21 

21 

22 

33 

11 

21.  3 

10 : 

30 

10 

29 

40 

13 

49 

31 

19 

20 

<-\  /*  ft 

26.9 

11 : 

10 

16 

11 

1 

25 

19 

25 

17 

^    A  T 

14.3 

12 : 

7 

8 

18 

11 

9 

18 

14 

11 

7 

11.4 

13: 

30 

2 

40 

30 

9 

42 

A  r? 

47 

52 

33 

31.7 

14: 

10 

1 

3 

14 

2 

31 

26 

8 

20 

12  .8 

15: 

28 

7 

33 

33 

19 

48 

36 

23 

34 

29.0 

lo : 

14 

14 

0 

23 

19 

25 

20 

28 

17 

ft  ft  rr 

20.7 

17  • 

22 

11 

7 

0 

9 

4 

1 

X 

ft 

2 

4 

PQ 

7 

o  .  o 

19: 

26 

11 

12 

10 

7 

10 

19 

20 

5 

13.3 

20: 

16 

20 

16 

37 

41 

39 

30 

50 

35 

31.5 

21 : 

11 

1 

1 

10 

6 

0 

ft  f\ 

20 

1 

5 

6.1 

22: 

15 

0 

1 

12 

5 

9 

37 

3 

8 

10.0 

23: 

3 

2 

5 

13 

6 

9 

10 

6 

15 

7.7 

24: 

28 

30 

20 

30 

16 

20 

40 

20 

31 

26.1 

25: 

6 

5 

5 

1 

5 

3 

10 

5 

0 

4.4 

26: 

7 

0 

5 

15 

5 

5 

14 

10 

17 

8.7 

27: 

9 

8 

15 

14 

17 

20 

20 

21 

12 

15.2 

Aver-; 

age  : 

17.0 

10.1 

16.3 

19.0 

13.1 

20.1 

25.5 

17.8 

18.2 

17.4 

12.0 

5.8 

11.7 

12.7 

10.2 

14.6 

10.0 

14.3 

13.4 

11.6 

1.6 

.8 

1.5 

1.6 

1.3 

1.9 

1.3 

1.9 

1.7 

1.5 

Table  XXVI  is  read  as  follows:     Tlie  mean  difference 


between  the  highest  and  lov/est  scores  on  Trait  I,  given  by  the 
three  Judges  to  the  27  instructors  was  17.0.    Sixty-eight  per 
cent  of  these  differences  fall  within  17.07  i:  12.0  or  between 
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5,0  and  29,0,    The  differences  on  Trait  I  range  from  3  to  40, 
as  may  "be  seen  by  inspection.    If  corresponding  data  were  de- 
rived from  other  groups  of  like  size  and  composition  the  chances 
are  even  that  the  mean  would  lie  within  17.0  i:  1»6  or  between 
15.4  and  18.6. 

A  study  of  Table  XXYI  reveals  a  large  spread  of  dif- 
ference between  the  scores  given  by  the  three  judges.    This  is 
particularly  true  of  certain  traits,  in  the  case  of  the  majority 
of  the  instructors,  and  of  all  the  traits  in  the  case  of  certain 
instructors.    It  must  not  be  forgotten,  however,  that  the  pos- 
sible range  of  difference  in  judgment  is  quite  large,  from  0  to 
100.    In  view  of  this,  the  disparity  in  the  scores  given  by  the 
judges  is  surprisingly  small  in  many  cases.     The  evident  con- 
clusion, hov/ever,  is  that  the  ratings  given  by  the  judges  are 
quite  subjective. 

Comparison  of  the  Scores  Given  by  Judges  and  Students. 

In  comparing  the  scores  given  by  judges  with  those  giv- 
en by  the  students,  use  was  made  of  both  the  correlation  method 
and  the  method  of  differences.    But  neither  of  these,  nor  both, 
permit  of  close  analysis  of  the  data.    Since  the  raw  data  are  not 
very  numerous,  they  are  presented  in  full  in  Table  XXYII. 

This  table  should  be  read  as  follov/s:    In  the  case  of 
the  first  instructor  the  mean  of  the  judges*  scores  on  Trait  I 
is  80,  the  mean  of  the  students*  scores  is  90,  making  a  differ- 
ence of  10  between  the  mean  scores  of  the  judges  and  the  mean 
scores  of  the  students* 
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TABLE  XXVII 

COR/rPARISON  BSTT/EEN  IffiAN  SCORE  OF  JUDGES  Ml)  ME^ 
SCORE  BY  STUDENTS  ON  EACH  TRAIT  FOR 
EACH  OF  27  INSTRUCTORS 


3  CJ 

H  <D  to 

:>  0)  CD 

^  ^  !m 

^  +3  O 

1  CD  O 


II  80  90  10  :  :  2  84  80  -4 

II  90  97  7  :  :  86  87  1 

III  82  91  9  :  :  84  82  -  2 

IV  78  87  9  :  :  80  89  9 

VI  77  91  14  :  :  93  81  -12 

VIII  73  83  10  ;  :  89  90  1 

IX  63  86  23  :  :  75  94  19 

X  78  91  13  :  :  88  91  3 

XI  78  90  12  :  :  83  96  13 

I        3  73  87  li  :  :  4  65  72  T 

11  85  92  7  :  :  63  68  5 

III  77  82  5  :;  :  47  64  17 

IV  67  82  15  :  :  47  79  32 

VI  69  77  8  ;  :  77  75  -2 

VIII  64  78  14  :  :  45  81  36 

IX  40  79  39  :  ;  48  86  38 

X  62  80  18  :  :  34  71  37 

XI  63  84  21  :  :  67  84  17 


1        5  74  81  7  :  :   6  46  88  42 

II  84  92  8  :  :  47  90  43 

III  64  66  2  :  :  30  76  46 

IV  73  68  -  5  :  :  33  82  49 

VI  84  78  -  6  :  :  69  83  14 

VIII  58  81  23  :  :  21  70  49 

IX  34  69  35  :  :  20  87  67 

X  63  71  8  :  :  19  76  57 

XI  67  64  -  3  :  :  33  81  48 


17  92  86  -  6  :  :  S        94  98  4 

II  89  86  -  3  :  :  97  98  1 

III  77  74  -  3  :  :  92  96  4 

IV  62  85  3  :  :  96  96  0 

VI  91  78  -13  :  :  96  91-5 

VIII  83  85  2  :  :  94  97  3 

IX  79  81  2  :  :  88  92  4 

X  79  82  3  :  :  91  98  7 

XI  81  84  5  :  :  92  98  6 
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TABLE  XXVII  (continued) 


0) 

o 
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'''The  minus  sign  indicates  that  the  mean  of  the  students  scores 
is  less  than  the  mean  of  the  judges  scores. 


Table  XXVII  shows  that  the  differences  between  the 
scores  given  by  the  judges  and  those  given  by  the  students  vary 
greatly  in  size,  as  between  traits  and  instructors.     These  dif- 
ferences are  the  significant  data  in  Table  XXVII  and  in  order 
to  make  a  closer  analysis  of  them  more  convenient  the  differ- 
ences between  the  judges'  scores  and  those  of  the  students  are 
shown  in  Table  XXVIII,  along  v/ith  the  standard  deviations  of  the 
distribution  of  the  differences  from  their  respective  means,  and 
the  probable  errors  of  their  means. 
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TABLE  X^IVIII 
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*The  minus  sign  indicates  that  the  mean  of  the  students'  scores 
is  less  than  the  mean  of  the  judges'  scores. 
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Table  XXVIII  should  be  read  as  follows:     The  differ- 
ences between  the  scores  given  by  the  judges  and  by  the  students, 
for  27  classes  on  Trait  I,  range  from  0  to  42,  with  a  mean  of 
8.29.     Sixty-eight  per  cent  of  these  means  lie  within  8.29. ±  8.80 
or  between  0  and  17,09.     The  chances  are  even  that  the  obtained 
mean  of  8.29  for  Trait  I  does  not  differ  from  the  true  mean  by 
more  than   +1.14.     Vi/'e  may  be  certain  (99  chances  in  100)  that  the 
true  mean  of  the  differences  between  the  judges'  scores  and  the 
students*    scores  on  Trait  I  lies  v/ithin  the  li  lits  8.29  *  4  x  1.14 
or  between  3.73  and  13.85 

It  should  be  noted  that  the  standard  deviations  of  the 
distribution  of  the  differences  are  larger,  in  the  case  of  many 
of  the  traits,  than  the  means  of  the  differences  between  the 
judges'  scores  and  the  students'  scores  on  the  sai.ie  traits.  This 
indicates  a  very  much  skewed  distribution  of  the  differences,  and 
an  inspection  of  Table  XXVIII  reveals  the  existence  of  a  few  dif- 
ferences, in  the  case  of  each  trait,  which  are  much  out  of  line 
with  the  majority  of  the  differences  and  with  the  mean.     The  ex- 
istence of  these  very  large  differences  tends  to  increase,  mater- 
ially, the  means,  the  standard  deviations,  and  the  probable  er- 
rors of  the  means.    L'nder  such  conditions,  and  especially  when 
the  data  are  not  very  nurnerous,'^  it  is  more  enlightening  to  study 
the  individual  scores. 

Referring  to  Table  XXVIII  it  will  be  noticed  that  on 


%arl  J.  Kolzinger.     Cp.  cit.  p. 79. 


-143- 

certain  traits,  i.e.,  IV,  VIII,  IX,  X,  and  XI,   the  disparity 
between  the  scores  of  the  judges  and  of  the  students  is  very 
wide.     On  Trait  IV  the  differences  for  six  instructors  ranged 
from  20  to  49,  with  a  mean  of  29;  on  Trait  VIII  six  differ- 
ences ranged  frora  20  to  49,  with  a  mean  of  35;  on  Trait  IX 
thirteen  differences  ranged  from  23  to  67,  v/ith  a  mean  of  37; 
on  Trait  X  six  differences  range  from  24  to  58  v/ith  a  mean  of 
40;  and  on  Trait  XI  four  differences  range  from  21  to  48  with 
a  mean  of  33. 

It  is  interesting  to  note  that  these  instances  of 
large  differences,  between  the  score  of  the  judges  and  that  of 
the  students,  are  confined  quite  largely  to  certain  instructors, 
i.e.,  those  numbered  6,  10,  14,  16,  20.    Moreover,  these  dif- 
ferences are  all  positive,  indicating  that  the  students'  scores 
in  all  cases  were  higher  than  those  of  the  judges.     This  might 
be  interpreted  as  evidence  of  the  influence  of  the  personality 
of  the  instructor  in  securing  for  himself  a  favorable  rating  in 
spite  of  his  lack  of  teaching  ability. 

It  should  also  be  observed  that  in  the  case  of  cer- 
tain traits,   i.e.,  I,  II,   III,  and  VI  for  all  ths  instructors, 
except  those  numbered  6,  16,  and  20,  the  differences  betv/een  the 
mean  scores  of  the  judges  and  of  the  students  might  be  regarded 
as  surprisingly  siiiall.     These  differences,  when  those  for  in- 
structors 6,  16,  ana  20  are  omitted,  ranged  from  0  to  14,  with 
a  mean  of  5.45  for  Trait  I;  from  0  to  18  with  a  mean  of  4.79 
for  Trait  II;  from  0  to  21,  v/ith  a  m-ean  of  7.16  for  Trait  III; 
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and  from  0  to  14,  with  a  mean  of  5.5  for  Trait  IV. 

In  the  case  of  the  majority  of  the  instructors,  i.e., 
those  remaining  after  nuiabers  5,  10,  14,  16,  and  20  are  ex- 
cluded, the  differences  between  the  mean  scores  given  by  the 
judges  and  by  the  students,  for  all  the  nine  traits,  are  seen 
to  be  small.    However,  the  wide  distribution  of  these  differ- 
ences between  the  judges*  mean  scores  and  those  of  the  stud- 
ents, as  indicated  by  the  standard  deviations  and  emphasized 
by  the  study  of  the  individual  differences,  demonstrates  clearly 
that,   in  the  rating  of  a  group  of  instructors,  if  the  mean 
score  of  the  judges  is  accepted  as  representing  accurately  the 
quality  of  instruction,  the  mean  scores  given  by  the  students 
will  often  fall  far  short  of  being  valid.     This  was  especially 
true  for  six,  or  22  per  cent,  of  the  27  instructors  studied. 

Because  of  the  li:;iited  number  of  judges  it  is  not 
5 

practicable    to  calculate  the  standard  deviation  of  the  means 
of  their  scores,  and  from  this  standard  deviation  and  the  stan- 
dard deviation  of  the  mean  scores  given    by  the  students,  ar- 
rive at  the  standard  deviation  of  the  difference  between  the 
tv/o  obtained  means,  for  each  trait.     This  makes  it  impracticable 
to  express,   in  terms  of  any  one  statistical  quantity,  tl'ie  re- 
liability of  the  differences  between  any  of  the  two  obtained 
means . 

In  an  attempt  to  apply  more  elaborate  statistical 
treatment  to  the  solution  of  this  problem  of  validity,  as  mea- 
sured by  the  correspondence  between  the  ratin  -s  of  experts  and 
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of  students,   correlations  were  run  between  the  mean  of  the 
judges'  scores  and  the  mean  of  the  students'  scores.     The  coef- 
ficient of  correlation  between  the  tv/o  scores  on  the  nine  traits 
for  each  individual  teacher  was  first  calculated  and  after  that 
the  coefficient  for  the  whole  group  of  27  instructors  v/as  com- 
puted by  Fisher's  method  of  averaging  coefficients  of  correla- 
tions.    The  numerical  values  of  these  coefficients  are  presented 
in  Table  XXIX. 

TABLE  XXIX 


CORRELATION  BET\iEEN  IvIEAi'J  OE  JTJDGES  SCORES  AND  MEAN  OF 
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.566 
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11 
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.224: 
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.223 

12 

-.228 

.213 

:  26 

.097 

.223 

13 

-.016 

.225: 

:  27 

.400 

.188 

14 
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.174: 

Average 

.255 

.183 

Table  XXIX  should  be  read  as  follows:    The  coefficient 
between  the  means  of  the  judges'  scores  and  the  means  of  the 
students'  scores,  on  each  of  the  nine  traits,  for  teacher  No.  1 
is  .796,    The  chances  are  even  that  the  "true"  r  falls  v/ithin  the 
limits  .796*. 081,  or  between  .715  and  .877.    It  is  practically 
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certain  (99  chances  in  100)  that  the  "true"  r  falls  v/ithin  the 
limits  .796  ±  4  x  .08  or  betv/een  .472  and  1.12.     By  the  "true" 
r  is  meant  that  r  v;hich  one  would  get  betv/een  the  mean  scores 
of  the  judges  and  the  students  on  every  member  of  the  whole 
population  of  college  instructors,  frora  which  instructor  No.  1 

a. 

is  presumably  dravm  at  random. 

The  correlation  method  of  measuring  relationship  is 
subject  here  to  the  saiae  weakness  noted  in  Chapter  IV  where 
the  question  has  been  more  fully  discussed.     In  other  words 
the  correspondence  betv/een  the  means  of  the  judges'  scores  and 
those  of  the  students  seems  to  this  investigator  to  be  greater 
than  the  numerical  value  of  the  coefficient  of  correlation 
between  them  would  indicate.     This  may  be  seen  clearly  by  com- 
paring the  two  sets  of  scores  for  each  of  the  instructors. 

It  is  interesting  to  note  in  this  connection  that 
the  coefficients  of  correlation  between  the  scores  of  the  judges 
and  of  the  students,  in  the  case  of  instructors  whose  nuiiibers 
are  6,  10,  14,  and  16  are  all  in  the.  vicinity  of  .5,  which  is 
considerably  above  the  mean  of  all  the  coefficients.  These 
are  the  classes  in  v/hich  the  large  differences  exist  between 
the  judges'  scores  and  those  of  the  studen\>s,     (See  Table 
ZXXVIII). 

It  must  be  admitted  that  the  results  of  this  attempt 
to  determine  the  validity  of  the  scale  are,  at  best,  unsatis- 

^  Henry  3.  Garrett,  Statistics  in  Psychology  anc  i!]ducation. 
Nev/  York:     Longmans,  Green  and  Go.     1926,  p.  170. 
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factory,  and  do  not  permit  any  definite  c  inclusion  to  be  i.iade 
regarding  it.  The  correlation  data  are  particularly  faulty,  due 
to  the  wealcnesses  alread;-  explained.  The  difference  method  does 
reveal  what  might  oe  regarded  as  a  high  coincidence  or  consider- 
able corres.:ondence  between  the  judges'  scores  and  the  students' 
scores  in  about  75  per  cent  of  the  classes. 


Chapter 

THE  SPREAD  OF  GE]^ERAL  ESTIMATE 
OR  "HALO  EFFECT" 

This  matter  has  already  been  referred  to  in  the  re- 
view of  literature,  where  conflicting  reports  from  different 
investigators  concerning  the  existence  and  extent  of  the  spread 
of  general  estimate  are  presented,  and  in  Chapter  lY  where  the 
technique  for  measuring  its  extent  is  discussed.    There  would 
seem  to  be  three  plausible  explanations  for  the  failure  of 
careful  investigators  to  discover  similar  findings.     In  the 
first  place,  the  different  investigators  worked  with  rating 
scales  varying  greatly  in  design  and  in  their  constituent 
items;  in  the  second  place,  the  subjects  being  rated  and  the 
raters  were  not  sufficiently  alike  in  the  different  studies  to 
make  the  results  comparable;  and  in  the  third  place,  the  in- 
vestigators lacked  any  specific  objective  standard  by  which  to 
interpret  their  findings  in  terras  of  the  existence  and  extent 
of  the  spread  of  general  estimate,"^    It  might  well  be  that  this 
tendency^  to  allow  one's  general  estimate  of  an  individual  to 
affect  the  rating  on  specific  traits  is  a  function  of  the 
measuring  device  used,  or  of  the  character  and  ability  of  the 
person  doing  zhe  rating.     In  other  words,  the  construction  of 

L.  Thorndike,  "A  Constant  Error  in  Psychological  Ratings," 
Journal  of  Applied  Psychology,  IV  {March,  1920)  29. 
^Walter  V.  Bingham  and  Max  Freyd,  Procedures  in  Employment 
Psychology.  New  York:  A.  W.  Shaw  and  Co.,  1926.  pp. 134-135. 
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a  given  scale,  or  a  careful  choice  of  raters,  or  both,  might 
tend  to  overcome  this  tendency,  which  is  probably  an  ever- 
present  danger  in  rating  practice,*-^ 

The  two  methods  of  measuring  the  spread  of  general 
estimate,  employed  in  this  investigation,  are  described  in 
Chapter  IV.    In  Table  XXX  the  evidence  derived  by  one  of  these 
methods,  i.  e»,  the  intercorrelation  method  is  presented.  One 
thousand  and  seven  scales  were  selected  at  random  from  the  8464 
scales  which  were  obtained  in  the  survey  made  in  the  winter 
quarter  of  1929.     Inter correlations  were  run  between  the  score 
on  the  different  traits.    The  obtained  coefficients  are  pre- 
sented in  Table  XXX. 

In  this  table  the  coefficients  of  correlation  are 
"raw",  that  is,  they  are  uncorrected  for  attenuation.    In  Table 
XXXI  the  corresponding  intercorrelation  coefficients,  corrected 
for  attenuation,  are  presented.    As  this  correction  always  in- 
creases the  numerical  values  of  the  coefficients,  those  report- 
ed in  Table  XXXI  are  greater  than  the  corresponding  coefficients 
in  Table  XXX.  ^ 

A  %^1 

The  formula*  used  for  this  correction  is  r.r>=-  ^  

in  which  r.-o  ~  coefficient  corrected  for  attenuation,  r.  ^ 

AB  AiBg, 

p 

^1-^2 »  ^"^^^  ~  coefficient  of  reliability  of"  the  different 

L.  Thorndike,  Op.  cit.  p.  28 
%enry  E.  Garrett,  Statistics  in  Psychology  and  Education.  New 
York:  Longmans,  Green  &  Co.,  1926.  pp. 212-213. 


-150- 


to 


tOCOlOCOJN'^OlO'^WtOo^cOOi 


CO 

E-f 
M 

<} 


EH 

PQ  CO 


o 

M 

o 

o 

M 

« 

o 
o 

K 


o 

CO 

o 
o 


o 


lO 


CO 


O 
H 


CO 


to 


ioa*">^c^coiO'*o>c\2i>-cvjcooOiO 

tOCQ'^^tOCOtOcONtOtO'^-^O'* 


0>  lO 


OlOCOtv-HO>HCOO'*OCOC\2 


lO  to  CO 


INtoO"«^C>-tOcv3HlO«OCOO 


W  CO  ^  £> 

lO  o  to 


OrH«3tOO>'>*t>-WO>«DOi 
'*tOlO';f«tO'*tOtOCO"<*'* 


O  CO  (M  O  CO 
to      "<;i<  'nH  to 


WHCOOWHcOCOOtO 

^^^^^T;j<tOtO"<ii<"«4< 


(35  00  O  Cvj  to 
CO  CM  to 


lOinowooorHW 

CO'*lO'^CV2';l<CVl^^ 


to  <0  «D  -"^  t>  W  to 
^  ^  si<  ^  to  to  CO 


tOWtOO>tDOCOtO 

cocococowtocviw 


^^COiMCOCOCOCO 


CVI  to  00  C>  CV3  H  to 
CO  CO  CO  CO  to  to  CV2 


tOHCOWHCOO>lOt> 
CV2vi<tOCOtOCOCv3CMCO 


W  CO  lO  to  to 
CO  'ii*  CO  to  to  ^ 


tOQOO^tOOCVJH-^OCO 
CV2(MCVICOtOCOvi<COsi«'?i4 


to  t>-  c\2  to 

CO  '^i*  lO  CV2  lO 


tOrHOOiH'sHtOOlO'^IN 
^HCOCO^i<«!i<'<4<'*lOCOtO'^ 


IN  W  CM  O 
CO  lO  lO 


^COCVltOOWOiC^HCOo-CO 
tOCOtOcaCvj'«;l<COCOtDCOWtO 


00  CO 
lf5  -"i*  lO 


otom£>-o>ojo>cO'<^«t>.ooof> 

IOC\2COCOCO^';1<'^CO'<^COWCO 


CO 


CV2 


j>  CO 


'^WWoOOtOOCOtOOCO'^lOO. 
CO?) 


to  lO 


coin">>HiOLOtO'*cocvj 


lO 


tOCOCVlc^tOtOCVllOHHtOlOCTiCOH 
lO^lO'^'^tOtO'^^'^lO'^lOCOtO'^ 


Ht>-COlStOCMCVIlOlOWOOOlOtOCO 
lO'^LOCOCOCOCOtOCO'>^';J<W^tOWCO 


CO 

EH 

M 


MMH      |>MMM  |x|HMMM>M 


-151- 

traits,  between  two  applications  of  the  first  scale,  and 

r,    m  the  coefficients  of  intercorrelations  between  the 
traits,  taken  from  Table  XXX,  The  coefficients  of  reliability 
used  in  calculating  these  corrections  were  those  obtained  in 
the  study  on  reliability  described  in  Chapter  VII.     It  should 
be  noted  that  the  429  ratings  used  in  securing  these  reliabil- 
ity coefficients  and  the  1007  used  in  obtaining  the  intercor- 
relation  in  Table  XXX  are  not  taken  from  the  sarae  identical 
group  of  raters.    But  there  is  no  essential  difference  in  the 
personnel  of  the  two  groups  of  raters,  both  being  composed  of 
students  in  classes  selected  at  random,  in  the  same  institu- 
tion. 

The  intercorrelation  coefficients  between  the  traits, 
when  corrected  for  attenuation  in  the  manner  described  above, 
are  set  forth  in  Table  XXXI.     It  will  be  noted  that  the  co- 
efficients have  been  increased  quite  materially  by  being  cor- 
rected for  attenuation,  but  there  is  still  an  open  question  as 
to  whether  these  coefficients  are  large  enough  to  serve  as  evi- 
dence of  the  existence  of  halo  effect.    In  the  absence  of  any 
objective  standard  as  to  how  large  these  coefficients  should  be 
in  order  tliat  they  may  be  interpreted  as  indicating  the  exist- 
ence of  halo,  it  77as  considered  worth  while  to  compare  the  co- 
efficients secured  in  this  present  study  with  the  findings  of 
other  investigators  of  this  problem.    The  datn.  for  the  Purdue 
Rating  Scale  for  College  Instructors  should  be  quite  comparable 
with  those  relating  to  the  College  Rating  Scale  designed  in  this 
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investigation,  since  they  both  are  based  upon  the  rating  of 

college  teachers  by  their  students,  and  conducted  under  sorae- 

what  similar  conditions.     In  Table  XXXII  the  intercorrelation 

coefficients  reported  for  the  Purdue  Rating  Scale  by  Stalmaker 
5 

and  Reimers    are  presented. 


TABLS  XXKII 

INTERCORRELATION  COSFFICIfiNTS*BETVffiEN  TRAITS  OF 
PURDUE  RATING  SCiiLE  FOR  COLLEGE  TEACHERS. 
CORRECTED  FOR  ATTEIWATION. 


Traits 

I 

II 

III 

IV 

V 

VI 

VII 

VIII 

IX 

X 

I 

.331 

.405 

.406 

.472 

.535 

,817 

.515 

.443 

.577 

II 

•  331 

.440 

.353 

.461 

.500 

.617 

.472 

.232 

.350 

III 

.405 

.440 

.403 

.588 

.528 

.350 

.514 

,306 

.525 

IV 

.406 

.353 

.403 

.306 

.514 

.438 

.091 

.248 

.306 

V 

.472 

.461 

.388 

.306 

.584 

.537 

.651 

,654 

,686 

VI 

.535 

.500 

.528 

.514 

.584 

.142 

.181 

.075 

.028 

VII 

.817 

.617 

.350 

.438 

..537 

.142 

.589 

.417 

.469 

VIII 

.515 

.472 

.514 

.091 

.651 

.181 

.589 

.599 

.549 

IX 

.443 

.232 

.306 

.248 

.654-. 075 

.417 

.599 

.496 

X 

.577 

.360 

.525 

.306 

.686 

.028 

.469 

.549 

.496 

Aver- 
age 

.500 

.418 

.451 

.341 

.549 

.326 

.485 

.462 

.369 

.444 

■sr  

All  coefficients  are  positive. 


It  may  be  observed  that  the  coefficients  reported  for 
the  Purdue  Rating  Scale  are  very  similar  in  size  to  those  ob- 
tained in  this  study.    The  mean  value  of  the  coefficients  for 
the  Purdue  Scale  is  .43,  while  for  the  College  Rating  Scale  de- 


J.  M,  Stalmaker  and  H.  H.  Remmers,  "Can  Students  Discriminate 
Traits  Associated  with  Success  in  Teaching?**  Journal  of  Ap- 
plied Psychology,  XII  (December,  1928),  610, 
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veloped  in  this  investigation  the  mean  of  the  coefficients  ob- 
tained is  •47, 

The  coraparison  between  the  coefficients  of  correla- 
tion obtained  in  these  two  separate  investigations  may  be  facil- 
itated by  presenting  the  data  in  the  form  shown  in  Table  XXXIII, 
in  which  the  coefficients  obtained  in  both  are  arranged  in  a 
frequency  distribution* 

TABLE  XXSCIII 

DISTRIBUTION  OF  IKTERCORRSLATION  COEFFICISNTS 
BETWEEN  TRAITS  IN  'THE  PURDUE  RATIl^^G  SCALE 
AlW  IN  THE  COLLEGE  RATING  SCALE. 


Numerical  values  of 
the  intercorrelation 
coefficients . 


Number  of  coefficients  in  each  intervaL 


For  Purdue  For  College 

Rating  Scale    Teachers  Rating  Scale 


-.01 

00 

-.10  ; 

:  1 

0 

.01 

.10  = 

3 

2 

.11 

.20  ' 

'  2 

0 

.21 

.30  '• 

'  4 

8 

.31 

•  40  ' 

7 

29 

.41 

.50  ' 

'  10 

39 

.51 

•  60  • 

'  13 

42 

.61 

.70  • 

•  4 

10 

.71 

.80  ' 

0 

2 

.81 

•  90  ' 

1 

4 

N  =45 

N  s  136 

Table  XXXIII  shows  that 
efficients  of  intercorrelation  for 
lege  Scale   are     quite  similar. 

Stalmaker  and  RemLiers^, 


the  distributions  of  the  co- 
the  Purdue  Scale  and  the  Col- 

who  report  the  data  concern- 


^Ibid.    p.  610. 
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ing  the  Purdue  Scale,  offer  it  as  evidence  that  the  "halo  ef- 
fect" in  the  Purdue  Rating  Scale  is  negligible.     The  writer 
does  not,  however,  venture  to  argue  from  his  data  on  this  point, 
that  the  "halo  effect"  is  either  negligible  or  significant,  be- 
cause, as  explained  in  Chapter  IV,  he  does  not  possess  any  ob- 
jective standard  or  measure  of  the  numerical  values  these  co- 
efficients of  intercorrelation  should  have  in  order  to  indicate 
the  presence  of  "halo  effect ♦"    Some  writers'^  seem  to  hold  that 
positive  coefficients  of  correlation,  no  matter  how  small,  are 
evidence  of  its  existence,  but  to  the  present  writer  this  con- 
tention does  not  appear  sound,  for  reasons  already  discussed  in 
Chapter  IV. 

Another  measure  of  the  existence  and  extent  of  "halo 
effect"  which  was  used  in  the  study,  is  based  upon  the  differ- 
ences between  the  highest  and  the  lowest  scores  on  the  reports 
given  the  individual  instructors.     These  reports,  one  for  each 
class,  give  the  mean  of  the  scores,  received  on  each  trait. 
The  lowest  mean  score  recorded  on  each  of  466  class  reports  was 
subtracted  from  the  highest  mean  score  on  the  same  report.  The 
difference  thus  obtained  indicates  the  extent  to  which  the  stu- 
dents discriminated  between  the  different  traits.    The  obtained 
differences  are  shown  in  Table  XXXTV. 

Table  XXXIV  should  be  read  as  follows:  There  were  17 
classes  in  v/hich  the  differences  between  the  highest  and  lowest 
mean  scores  on  all  the  seventeen  traits  fell  within  1  -  10; 

19  — — —  

Edward  L.  Thorndike.  Op,  cit.  p,  28. 
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TABLE  XXXIV 


DIFFERENCES  BETViEEN  THE  HIGHEST  AND  LOYffiST 
MEAN  SCORES  GIVEN  BY  EACH  OF  456 
CLASSES,  7/INTEP.  1929. 


Range  of  differences  Number  of  classes 


1-10  17 

11  -  20  161 

21  -  20  163 

31  -  40  82 

41-50  26 

51  -  60  14 

61  -   3 

1  -  7G  466 


there  were  161  classes  where  the  differences  fell  within  11  - 
20,  etc. 

The  median  score  falls  in  the  21-30  group,  at  23.38, 
This  means  that  in  fifty  per  cent  of  the  466  classes  the  differ- 
ence betv/een  the  means  of  the  scores  given  on  the  highest  traits 
and  the  means  on  the  lowest  traits  was  below  23,38,  while  in 
the  remaining  fifty  per  cent  it  was  above  23.36.    Y/hether  this 
difference  is  significant  in  indicating  the  presence  and  extent 
of  "halo  effect'*  is  as  difficult  to  determine  as  in  the  inter- 
correlation  method. 

It  is  significant  that  in  at  least  a  small  proportion 
of  the  classes  the  obtained  differences  might  be  regarded  as 
being  large,  indicating  that  the  "halo  effect"  does  not  operate 
extensively,  if  at  all,  in  those  particular  classes.  This 
raises  the  question  as  to  7/hether  it  is  operative  in  the  cases 
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where  much  smaller  differences  are  found,  since  it  might  well 
be  argued  that  many  teachers  in  such  a  large  group  as  is  in- 
volved in  this  study  would  possess  all  the  traits  on  the  scale 
to  a  rather  large  extent,  and  that  this  would  account  for  the 
small  differences  recorded  in  many  of  the  classes.    In  other 
words,  siiiall  differences  between  the  scores  on  high-  and  low 
traits  may  be  explained  on  other  grounds  than  that  of  "halo 
effect."    On  tliis  basis  one  might  well  reason  that  the  "halo 
effect"  does  not  operate  to  any  very  serious  extent  in  the  Col 
lege  Teachers  Rating  Scale  developed  in  this  investigation. 


CHAPTER  X 

CERTAIN  MISC3LLANE0US  PR0BLEi..!i3  RELATED  TO  THE  USE 
OF  THE  COLLEGE  TEACHiCRS  RATING  SCALE 

Although  the  main  objective  of  this  investigation 
is  the  determination  of  the  reliability  and  validity  of  the 
Iowa  State  College  Rating  Scale  for  Teachers,  certain  other 
problems,  connected  with  its  use,  were  brought  into  focus  and 
challenged  the  mind  of  the  investigator.     Only  four  of  these 
have  been  followed  far  enough  to  be  reported  upon  at  this 
time.     These  four  problems,  7;hich  are  discussed  in  this  chap- 
ter, nay  be  stated  briefly  as  follows. 

1.  »Vhat  relation  exists  between  the  ability  of  the 
student  and  the  rating  he  gives  his  instructors? 

2.  V/hat  relation  exists  between  the  maturity  of  the 
students  anc  the  rating  v/hich  they  give  their  in- 
structors? 

3.  \'{hat  relation  exists  between  the  size  of  classes 
ana  the  average  ratings  received  by  their  instruc- 
tors? 

4.  V/hat  is  the  rrend  in  the  nui:.erical  values  of  the 
scores  reosived  by  individual  instructors  in  suc- 
cessive ratings? 

These  four  problems  vvill  now  be  discussed  in  the 
order  listed  above.     The  method  of  investigating  these  problems, 
the  statistical  findings  and  the  conclusions  which  seem  to  be 

-158- 


-159 


warranted  will  be  subLiitted  in  soi.ie  aetail. 

1.  vVhat  Relation  iixists  between  the  Ability  of  the 
Student  and  the  Rating  he  Gives  His  Instructor? 

This  problem  was  suggested  by  the  fact  that  several 
instructors  objected  to  being  rated  by  their  duller  students, 
while  professing  to  be  perfectly  willing,  and  even  eager,  to 
submit  to  a  rating  by  their  better  studen:s.     The  v/riter  was 
enabled  to  nake  a  comparison  of  the  rating  given  by  each  stu- 
dent in  19  classes  containing  443  students,  with  th.  grade 
given  to  each  student  by  his  instructor.     The  grade  given  by 
the  instructor  to  the  student  was  taken  as  a  measure  of  his 
ability. 

The  same  classes  were  used  as  in  the  reliability  study 
already  reported  on  page  114,  together  v;ith  two  additional  clas- 
ses not  included  in  tht  study  of  reliability.    Each  student  had 
been  assigned  a  nuriiber  v/hich  he  wrote  upon  the  scale  on  which  he 
recorded  his  opinion  of  his  instructor,     xi.  record  of  the  names, 
with  their  respective  nu-.bers,  was  kept  anc.  after  the  o/aarter 
was  over  anc  the  grades  had  been  reported  to  the  registrar  it 
was  possible  to  secure  a  copy  of  the  report  for  each  class. 

Correlations  were  run  between  the  ratixigs  given  by 
each  student  to  his  instructor  and  the  grade  reported  by  the 
instructor  for  tne  saiae  student.     The  coefficients  obtained  are 
listed  below. 

Table  XXXV  should  be  read  as  follows:     The  coeffic- 
ient of  correlation  between  the  rating  given  by  each  of  the  18 
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students 

in  the 

first  class,  and  the  grade  given  to 

the 

sarae 

student 

by  th^ 

irLS"cruc-cor  of  this  class  is     .1021  ir 

.149 

• 

The  chances  are 

even  that  the  "true"  coefficient  of 

correlation 

falls  within  th 

0  limits  .1021  "jr  .149,  or  between  -.047  and 

.2511. 

It  is  practically  certain  (the  chances  ^re  9 

9  in 

100) 

that  the 

"true 

coefficient  falls  vathin  the  limits 

.1021 ± 

4  X  .149 

or  betv;een  -.494  and  .698. 

TABLE  XXKV 

CORRELATION 

BETWEEN  TH^L  SCORE  BITEK  TO  INSTRUCTOR 

BY  : 

EACH 

STUDi^HT 

AI\^D  GRADE  GIVEN  TO  HIM  BY  SAIvIE  INSTRUCTOR 

Number  of 

Class 

Students  r 

PEr- 

i. 

1 

18  .1021 

.149 

2 

21  ,3089 

.137 

3 

17  .0577 

.163 

4 

21  .3524 

.119 

5 

19  .2027 

.144 

6 

18  .4231 

.125 

7 

14  .0472 

.178 

8 

29  .0529 

.122 

9 

22  .0027 

.150 

10 

46  .0989 

.099 

11 

22  .1035 

.133 

12 

38  .0368 

.106 

13 

38  .1014 

.105 

14 

31                                 .137  0 

.120 

15 

35  .0558 

.113 

16 

29  :^121 

.112 

17 

17  .4152 

.136 

18 

34  .2463 

.107 

19 

32  .0963 

.121 

Average 

26.3  .1545* 

.031 

*This  average  r  v/as  obtained  from  the  r's  of  the  classes  by 
the  use  of  the  formula  suggested  by  R.  A.  Eisher.     This  for 
mula  has  already  been  referred  to  on  page  55. 
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From  the  data  presented  in  the  foregoing  table  it 
seems  safe  to  conclude  that  the  instructors  v/ho  objected  to 
being  rated  by  their  dull  students  have  very  little  statistical 
evidence  to  support  them.     In  the  majority  of  cases  the  coef- 
ficients of  correlation  are  considerably  smaller  than  their  re- 
spective probable  errors  and  in  no  case  is  the  coefficient  four 
times  the  size  of  its  PE.     It  is  generally  held  by  statisticians 
that  in  order  to  be  reasonable  sure  that  there  is  some  correla- 
tion present  an  obtained  coefficient  of  correlation  should  be  at 

1 

least  four  times  its  P2.     By  this  standard  our  obtained  coeffic- 
ients may  be  regarded  as  indicating  an  almost  entire  lack  of  cor- 
relation. 

Practically  identical  results  were  obtained  in  a  study 
conducted  by  the  statistical  service  of  the  Mathematics  Depart- 
ment of  Iowa  State  College.     This  study  was  never  published  but 
the  persons  who  conducted  it  have  announced  that  no  significant 
correlation  could  be  obtained  between  the  grades  given  to  stu- 
dents in  the  courses  of  this  department  and  the  ratin^^^s  given 
by  the  sarae  students  to  their  instructors. 

The  same  conclusion  has  been  reached  by  Reramer^  as  a 
result  of  a  study  made  at  Purdue  University.  Ke  reports  "there 
is  no  noticeable  tendency  for  students  to  vary  their  ratings  of 

Henry  E.  G-arrett,  Statistics  in  Psychology  and  Education. 
New  York:     Longiaans,  Green  and  Co.  1923,  p. 170, 

2 

H,  H.  Remmers,  The  eollege  Professor  as  the  Student  Sees  Him. 
Studies  in  Higher  Education  XI.     Purdue  University  Bulletin, 
Vol.  XXIX,  Ko.  6.  Lafayette,  Indiana:     Purdue  University,  1929. 
p. 63. 
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an  instructor  with  the  kind  of  marks  they  receive  from  the  in- 
structor.   The  average  correlation  between  ratings  and  marks 
for  all  traits  is.. 070 — practically  zero. 

The  failure  of  the  coefficient  of  correlation  to  mea- 
sure the  consistency  existing  between  two  sets  of  variables  when 
the  exl-i't  of  varie/^-ipT-  within  each  series  is  small,  has  already 
been  pointed  out  in  Chapter  IV.    An  inspection  of  the  raw  scores 
from  which  the  data  in  Table  XXXV  were  derived  indicate  that 
this  factor  operated  to  lower  the  coefficient  of  correlation  ob- 
tained.    In  other  words,  in  a  few  classes  the  scores  received 
by  the  instructor  from  each  student  and  the  grade  given  to  the 
same  student  by  the  instructor  were  very  similar  but  the  corre- 
lation between  them  v/as  very  small. 

This  condition  is  shovm  more  clearly  by  studying  the 
differences  between  the  scores  given  to  each  instructor  by  a 
student  and  the  grade  of  the  same  student.     These  differences 
are  presented  in  the  following  table. 

Table  IGQCYI  should  be  read  as  follov^s:     For  class  No. 
1,  with  18  students,   the  mean  of  the  differences  between  the 
score  given  *to  the  instructor  by  each  student  and  the  grade 
given  to  the  saiie  student  by  his  instructor  is  14.33.  Sixty- 
eight  per  cent  of  these  differences  fall  within  14.331:  11.04, 
or  between  5.??9  and  So. 37.        The  chances  are  even    that  if 
corresr)onrling  data  were  secured  for  other  classes  of  like  size 
and  composition,  the  means  of  the  differences  would  lie  within 
14.33      1.75  or  between  16.08  and  12.58.     It  is  practically 
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certain  that  the  true  mean  of  the  differences  lies  within  the 
limits  14.33  ±  4  x  1.75  or  between  7.33  and  21.33. 

TABLE  iCKXVI 


DIFFSRSi:CES  BETVSSN  THE  RATINGS  GIVEN  TO  INSTRUCTOR  3Y  EACH 
STUDENT  Aim  TKK  GRADE  GIVEN  TO  HIM  BY  HIS  INSTRUCTOR 


Lie  an  oi 

iDjj  OI  one 

O  T  o  o  o 

Ol 

Diiierences 

jjisiiriD  union 

^^m 

»D  u  uuen  b  s 

OI  Qiiieiences 

1 

lo 

14.  OO 

11 . 04 

1  .  /  0 

dl 

D     O  05 
O.  <CO 

0 .  y  U 

Q  C 
.  OD 

o 

1  / 

O  ,  4± 

4b.  d4 

.  /o 

4 

21 

10.52 

6.56 

.96 

0 

/I  Oft 
4t  .  (CD 

6 

18 

3.55 

2.81 

.44 

7 

1  4. 

no 

8 

29 

6.72 

3.64 

.45 

9 

22 

4.86 

3.38 

.48 

10 

46 

5.32 

3.87 

.38 

11 

21 

6.85 

5,02 

.74 

12 

38 

8.18 

7.19 

.79 

13 

38 

5.97 

4.75 

.52 

14 

31 

8.74 

6.27 

.76 

15 

35 

5.91 

3.66 

.41 

16 

29 

6.93 

3.85 

.48 

17 

17 

3.82 

2.63 

.43 

18 

34 

6.32 

4.60 

.53 

19 

32 

15.58 

19.41 

2.31 

Average 

26.3 

6.37 

5.55 

.74 

The  corresponding  data  for  the  other  classes  should 
be  read  in  the  same  manner.     V/hen  this  is  done  it  is  seen  that, 
except  for  classes  1,  4,  and  19,  the  mean  differences  between 
the  instructor's  ratings  ana  the  students'  grades  are  not  so 
large  as  might  be  expected  from  the  correlation  data  in  Table 
XXXV.     In  fact,  the  small  numerical  values  of  the  mean  differ- 
ences when  taken  alone  would  indicate  a  large  degree  of  corre- 
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spondence  between  the  two  factors  under  consideration.  But 
the  rather  large  values  of  tije  standard  deviations  of  the  dis- 
tributions of  the  differences,  when  compared  with  the  respec- 
tive means,  indicate  a  wide  range  in  the  size  of  these  differ- 
ences and  suggest  that  in  the  cases  of  many  individual  students 
the  differences  may  be  considerably  larger  than  the  means  of 
the  differences. 

For  instance,  in  class  2,  which,  from  the  size  of  its 
mean  difference  and  standard  deviation,  may  be  taken  as  an  ex- 
ample of  the  average  class,  sixty-eight  per  cent  of  the  21  in- 
dividual differences  obtained,  in  this  class,  fall  v/ithin  the 
limits  8,23 ±  5.90,  or  between  2.53  and  14,13,  and  practically 
all  the  differences  will  fall  within  the  limits  8, 23  ±  3  x  5,90, 
or  between  0  and  25,93.     For  the  majority  of  the  classes  the 
mfians  of  the  differences,  and  the  standard  deviations  are  con- 
siderably less  than  the  averages  for  all  the  classes,  due  to  the 
existence  of  classes  1  and  19  for  which  the  mean  difference  and 
standard  deviations  are  much  larger  than  for  the  others.  For 
exaraple,  in  class  10,  which  represents  the  classes  v/ith  the  small 
differences,  the  mean  of  the  differences  is  5.32,  Sixty-eight 
per  cent  of  the  46  individual  differences  fall  within  5,32  ± 
3,87,  or  between  1,45  and  9.19,  and  practically  all  the  differ- 
ences will  fall  between  zero  and  16.93, 

The  probability  of  v;ide  divergence  between  the  student  *s 
rating  of  his  instructor  and  the  academic  grade  given  to  the  stu- 
dent by  the  same  instructor,   in  at  least  a  number  of  cases,  is 
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apparent  from  such  an  interpretation  of  the  data  in  Table 
XXXYI.     This  is  true  even  though  the  mean  difference  for  the 
class  appears  to  be  small.     In  fact,  an  inspection  of  the  raw 
scores  reveals  the  existence  of  a  considerable  number  of  dif- 
ferences, in  the  case  of  individual  students,  which  are  much 
larger  than  the  mean  difference  for  the  whole  class.     It  is 
also  revealed  that  these  (Jifferences  are  not  in  the  same  direc- 
tion, that  is,  in  the  case  of  some  students  the  grades  are  higher 
thaEi  the  ratings  he  gives  his  instructor  v;hile  in  the  case  of 
others  the  reverse  is  true.     The  existence  of  these  large  dif- 
ferences are  indicative  of  considerable  lack  of  correspondence 
between  the  students'  grades  and  the  teachers'  ratings,  enough, 
probably  to  account  in  some  measure,  for  the  low  coefficients 
of  correlation  obtained. 

In  view  of  the  evidence  submitted  it  seems  reasonable 
to  conclude  that  the  ability  of  the  student,  as  measured  by  his 
academic  grades,  is  not  a  function  of  the  rating  v/hich  he  gives 
his  instructor. 

In  studying  Table  XXXVI  it  should  be  kept  in  mind  that 
the  theoretical  limits  of  the  rating  scale  and  the  grading  sys- 
tem used  at  Iowa  State  College  are  identical,  i.e.,  0  -  100, 
Moreover,  the  all-college  average  for  students'  grades  is  ap- 
proximately 84,  which  is  not  much  less  than  the  average  of  the 
scores  for  all  the  college  teachers  involved  in  this  investiga- 
tion.   Also  the  actual  range  limits  of  the  instructors'  scores 
and  the  students'  grades  are  quite  similar,  being  approximately 
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50  -  100.     These  similarities  between  teachers*  ratings  and 
students*  grades  make  it  possible  to  compare  these  two  variables 
directly  by  subtracting  one  from  the  other  to  obtain  the  dif- 
ferences between  them.     This  is  the  method  by  which  the  data  in 
Table  XXXVI  were  secured. 

S.  VThat  Relation  Exists  between  the  Maturity  of  the 
Student  and  the  Ratings  which  He  Gives  His  In- 
structors? 

This  became  a  live  problem  when  the  instructors  began 
comparing  their  respective  ratings,  since  many  of  then  taught 
the  freshmen,  ¥/hile  others  taught  upper  classm.en  exclusively. 

It  was  claimed  by  many  instructors  that  the  ratings 
made  by  the  different  classes  were  not  directly  comparable,  one 
with  the  other,  because  of  the  difference  in  maturity  of  judg- 
ment between  the  lower  and  upper  classmen. 

In  the  winter  (19S9)  survey  the  ratings  given  by  the 
four  different  years  had  been  kept  separated  and  further  sta- 
tistical treatment  was  given  these  ratings  in  the  attempt  to 
determine  what  difference,  if  any,  exists  between  the  ratings 
of  the  lower  and  upper  classes.     The  findings  are  presented  in 
the  tables  immediately  following. 

Table  XXXVII  shows  the  mean  score,  the  standard  de- 
viation of  the  distribution  of  the  individual  scores  from  the 
mean,  and  the  probable  error  of  the  obtained  mean  on  each  trait, 
for  each  of  the  four  college  years. 

Table  XiCXVII  should  be  read  as  follows:     The  mean 
scores  given  by  the  561  senior  students  on  Trait  I  is  80.5. 
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Sixty-eight  per  cent  of  the  individual  scores  on  Trait  I  fall 
between  80. b  ±  15.3,  or  betv/een  65. £  and  95.8.     It  is  practic- 
ally certain  that  all  the  scores  will  fall  between  80.5 ±  3  x 
15,3,  or  between  34.6  and  100.0,     if  ratings  v^ere  secured  from 
other  groups  of  like  size  and  composition  the  chances  are  50 
in  100  that  the  means  for  the  different  groups  would  fall  with- 
in 80.5  ±  .435,  or  between  80*065  and  80.935.     it  is  practic- 
ally certain  that  the  true  mean  lies  within  78.76  and  82. E4. 
The  corresponding  data  for  the  other  traits  and  the  other  clas- 
ses are  read  in  the  same  way. 

An  inspection  of  Table  XXXVII  reveals  the  existence 
of  considerable  similarity  or  correspondence  betv^een  the  rat- 
ings given  by  the  four  different  college  years.    This  corre- 
spondence may  be  more  definitely  expressed  by  calculating  the 
coefficient  of  correlation  between  the  means,  on  the  different 
traits,  given  by  each  pair  of  the  four  college  classes.  The 
coefficients  so  obtained  are  shown  in  Table  XXXVIII. 

TABLE  XXXVIII 

CORRELATION  BETV^EEN  THE  MEAI^I  SCORES,  GIVEN  BY  THE  FOUR 
COLLEGE  YEARS,  ON  EACH  OF  THE  TRAITS  ON  THE  SCALE 

Classes  Compared  r  PEp 


Senior  Junior  .77  .068 

Senior  Sophomore  .90  .032 

Senior  jj'reshmen  .90  .032 

Junior  Sophomore  .90  .032 

Junior  i'*reshmen  .90  .032 

Sophomore           Freshmen  .94  .019 

All-college—  Senior  .93  .022 

All-College—  Junior  .90  .032 

All-college —  Sophomore  .96  .013 

All-College —  jj'reshmen  ^90  »052 

Averages  »0Z1 
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I'able  XXXVIII  should  be  read  as  follows:     The  coef- 
ficient of  correlation  betv;een  the  mean  score,  on  each  of  the 
seventeen  traits  on  the  scale,  given  by  the  senior  year  stu- 
dents, and  the  corresponding  scores  given  by  the  junior  year 
students  is  .77,     If  the  scores  from  other  groups  of  like  size 
and  composition  were  correlated,  the  chances  are  50  in  100  that 
the  coefficients  thus  obtained  would  lie  v/ithin  .77  +  .068,  or 
between  .702  and  .838,     it  is  practically  certain  that  the  true 
coefficient  lies  within  ,77 ±   4  x  .068,  or  between  ,498  and 
1.00.     The  corresponding  data  for  the  other  comparisons  are 
read  in  the  same  manner. 

Table  XXXVIII,  when  studied  in  connection  with  Table 
XXXVII,  presents  clear  evidence  of  the  dose  correspondence  in 
the  ratings  given  by  the  four  college  years.     The  comparatively 
low  coefficient  obtained  between  the  ratings  by  the  senior  and 
the  junior  classes  may  be  explained  by  the  failure  of  the  cor- 
relation method  to  measure  the  correspondence  under  certain  con- 
ditions.   This  has  been  already  referred  to  in  Chapter  IV.  A 
comparison  of  the  mean  scores,  per  trait,  given  by  these  two 
classes  (See  Table  xXXVII)  reveals  a  large  degree  of  correspon- 
dence between  them. 

Another  method  of  expressing  the  correspondence  or 
coincidence  between  the  ratings  given  by  each  of  the  four  col- 
lege years  or  classes,  consists  in  subtracting  the  mean  scores 
given  by  one  class  from  the  corresponding  scores  given  by  an- 
other.    The  differences  thus  obtained  between  the  different 
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pairs  of  classes,  along  with  measures  of  their  statistical 
significance,  are  presented  in  detail  in  Table  aXXIX. 

Table  XXXIX  should  be  read  as  follov/s:     The  differ- 
ence between  the  mean  of  the  scores  given  by  1568  sophomores 
and  the  mean  of  the  scores  given  by  2S6S  freshmen  for  rrait  I 
is  .6.     The  chances  are  50  in  100  that  the  obtained  difference 
of  .6  does  not  diverge  from  the  true  difference  by  more  than 
±  .379.     The  chances  are  99  in  100  that  the  obtained  differ- 
ence of  .6  does  not  diverge  from  the  true  difference  by  more 
than  4  x  .379  by  more  than  1,516.    The  chances  are  86  in  100 
that  the  true  difference  is  greater  than  zero. 

When  the  corresponding  data  for  the  other  traits  and 

other  comparisons  were  interpreted  in  the  manner  illustrated 

above,  the  differences  betv/een  the  ratings  given  by  the  four 

separate  college  years  are  seen  to  be  inconsiderable,  with  but 

very  few  exceptions,  Yet  real  differences  do  exist  and  in  some 

cases  may  be  regarded  as  significant,    ivioreover  in  the  majority 

of  instances  the  chances  are  high  that  the  true  difference  is 

Dif  f 

greater  than  zero,     in  31  per  cent  of  the  cases  the  -dt?^  .  1^ 

-t  Jiiaii  f  ♦ 

is  large  enough  to  be  as  large  or  larger  than  the  statistical 
criterion  of  a  real  difference,  i.e.,^»     In  most  cases,  the 
differences,  though  real,  are  not  very  large. 

A  further  observation  &&y  be  made  of  the  fact  that 
there  exists  a  well-marked  tendency  on  the  part  of  the  sophomores 
to  rate  their  instructors  higher  than  do  the  other  classes.  The 

^  Henry  E.  Garrett.  Statistics  in  Psychology  and  Education. 
New  iiork:    Longmans,  Green  and  Go.,  1926 , pp. 133-136, 
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juniors,  on  the  contrary,  tend  to  rate  their  instructors  lower 
than  do  the  other  three  classes.    An  explanation  of  this  ten- 
dency will  not  be  attempted. 

jj'rom  these  data,  presented  in  Tables  XXXVIII  and 
XXXIX,  it  may  safely  be  concluded  that  the  differences  between 
the  ratings  given  by  students  in  the  different  college  years 
are  small  enough  to  be  disregarded  when  comparing  the  mean  class 
ratings  given  to  instructors  by  classes  in  the  four  college 
years.     It  should  be  kept  in  mind,  however,  that  in  only  a  few 
cases  were  the  same  instructors  rated  by  both  the  upper  and  low- 
er classes,  and  it  is  generally  admitted  that  it  is  the  custom 
to  assign  to  fresiinen  and  sophomore  classes  the  younger  and  less 
experiences,  and  presumably  less  effective,  members  of  the  fa- 
culty,    if  this  admission  be  accepted  as  a  fact,  it  might  be  re- 
garded as  evidence  that  the  freshmen  and  sophomores  tend  to  be 
less  critical  oi  the  quality  of  the  instruction  which  they  re- 
ceive than  are  the  upper  classes,  since  the  ratings  given  by  the 
first  two  years  are  equal  to  or  higher  than  those  given  by  the 
last  two  years.    Evidence  of  positive  correlation  between  exper- 
ience and  academic  training,  on  the  one  hand,  and  teaching  ef- 
fectiveness, on  the  other,  seems  to  be  lacking  and,  because  of 
this,  we  need  not  pursue  this  line  of  reasoning  farther. 

Remmers,  in  a  study  of  the  rating  of  college  teachers 
at  Purdue  university,  obtained  results  very  similar  to  those 
which  have  just  been  submitted,    referring  to  the  comparison  be- 
tween the  ratings  given  by  the  different  years,  he  asserts  that 
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"the  coincidence  of  standards  and  judgments  indicated  by  the 
data  are  far  more  striking  than  the  differences  between  them." 
This  conclusion  would  seem  to  the  fully  warranted  by  the  find- 
ings of  both  the  Purdue  study  and  the  one  reported  in  this  study. 

3.  What  Relation  Exists  between  the  Size  of  Classes 
and  the  Ratings  Received  by  Their  Instructors? 

In  the  solution  of  this  problem  an  attempt  was  made 
to  eliminate,  or  hold  constant,  as  many  as  possible  of  the  var- 
ious factors,  other  than  class  size,  which  might  influence  the 
rating  by  any  class.     To  this  end  classes  taught  by  the  same  in- 
structor in  the  same  course  or  subject,  and  in  the  same  term 
were  first  compared.     This  basis  of  selection,  however,  did  not 
produce  what  was  considered  to  be  a  sufficient  number  of  compar- 
isons.   As  it  had  already  been  demonstrated  (Chapter  VII  p.  127) 
that,  on  the  average,  the  ratings  received  by  any  given  instruc- 
tor were  not  differentiated  on  the  basis  of  the  different  courses 
or  subjects  taught  by  him,  it  was  decided  to  compare  classes  in 
different  courses  taught  by  the  same  instructor  during  the  same 
term,  and  to  combine  the  differences  thus  obtained  with  those  de- 
rived from  a  comparison  between  the  classes  in  the  same  courses. 
Even  with  this  combination  it  was  impossible  to  secure 
many  cases  in  the  high  and  low  sized  groups,  since  it  is  a  policy 
of  Iowa  State  College  to  discourage  the  formation  of  classes  under 
10  and  over  40.    For  this  reason  Table  XL  does  not  show  all  the 


H.  H.  Remmers.  Op.  cit.  p. 40. 
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possible  comparison  between  the  various  groups  as  it  was  arbi- 
trarily decided  not  to  include  tJriose  comparisons  containing 
less  than  12  cases. 

The  following  size  groups  were  used  in  comparing  the 
class  ratings: 
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Jilach  class,  railing  in  any  one  oT  the  size  groups  was 
paired  with  a  class,  in  another  size  group,  taught  by  the  same 
instructor,  during  the  same  term,     ihe  mean  scores  on  the  corre- 
sponding traits  in  each  pair  of  classes  were  then  subtracted. 
When  any  mean  score  given  by  tne  larger  class  was  smaller  than 
the  corresponding  score  given  by  the  smaller  class,  a  minus  sign 
was  placed  before  the  niimber  representing  the  difference  between 
them.    The  differences  thus  secured  for  eacn  group  of  classes 
compared  were  added  algebraically  and  the  mean  difference  for 
the  group  computed. 

The  mean  differences  thus  obtained  are  presented  in 

Table  XL. 

using  the  first  item  as  an  exaiaple,  Table  XL  is  to  be 
read  as  follows:    The  mean  of  the  differences  on  Trait  i  between 
the  mean  scores  given  by  18  classes  containing  from  1  to  6  stu- 
dents, and  the  corresponding  scores  given  by  tne  same  number  of 
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classes  containing  from  7  to  12  students,  to  the  same  instruc- 
tor, during  the  same  term,  is  b.bb.    Jixcept  in  the  case  of  the 
comparison  made  between  classes  in  size -groups  1  to  6  and  7  to 
12,  which  is  the  first  comparison  shown  in  rable  XL,  the  nean 
differences  between  ratings  made  in  classes  in  the  various  size 
groups  are  snown  to  be  small,    xhe  data  indicate    that  ratings 
received  from  classes  ranging  in  size  from  7  to  12  are  likely 
to  be  higher  than  in  classes  of  from  1  to  6  students.  Although 
the  differences  obtained  in  the  other  comparisons,  shown  in  Ta- 
ble XL,  are  probably  small  enough  to  be  disregarded  when  compar- 
ing the  ratings  of  teachers  by  classes  of  varying  sizes,  there 
seems  to  be  a  slight  tendency  for  the  ratings  to  increase  with 
class  size,  froiu  the  first  to  the  third  size-groups,  and  to  de- 
crease v/itn  class  size  in  the  remaining  size-groups,     if  one  v/ere 
required  to  decide  upon  the  optimum  number  in  a  class  from  these 
data  probably  20  should  be  tne  number  chosen,     it  must  be  observ- 
ed, however,  that  the  coincidence  between  the  ratings  for  the 
different  groups  is  much  more  striking  than  the  differences. 
The  evident  conclusion  to  draw  from  these  data  is  that  the  size 
of  a  class,  except  in  the  case  of  classes  with  less  than  seven 
students,  may  be  disregarded  in  comparing  ratings  given  to  in- 
structors by  classes  of  varying  sizes. 

•j^hese  findings  coincide  very  closely  although  the  dif- 
ferences are  on  the  whole  smaller,  with  those  discovered  by 

Brandenburg  and  i<emmers  in  their  investigation  with  the  i^urdue 
xiating  scale,     rhey  report  that  -these  differences  related  to 
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size  of  class  are  relatively  quite  unimportant  as  compared  with 
the  similarities  observed."^ 

4.  What  is  the  Trend  in  the  :\iuiiierical  Values  of  the 
Scores  Received  by  individual  Instructors  in  Suc- 
cessive Katings? 

The  ansv/er  to  this  question  was  attempted  by  comparing 
the  scores  received  by  individual  instructors  from  classes  taught 
during  successive  terms.     The  classes  were  compared  directly  by 
subtracting  the  mean  class  scores  per  trait,  given  by  one  class, 
from  the  corresponding  scores  given  by  a  second  class  in  another 
term.    When  the  scores  given  by  the  second  class,  in  point  of 
time,  were  smaller  than  the  corresponding  scores  given  by  the 
first  class  a  minus  sign  was  prefixed  to  the  difference.  The 
differences  thus  obtained  for  the  various  comparisons  made  were 
added  algebraically  to  yield  the  quantities  presented  in  the  ta- 
bles which  follow,    \yhere  two  or  more  classes  of  the  same  subject 
were  taught  by  the  same  instructor  in  the  same  term,  the  mean  of 
all  the  scores  given  by  these  classes  on  each  trait,  v/as  calculat- 
ed and  used  in  the  comparisons  referred  to  above. 

Table  XLI  shows  the  differences  obtained  between  rat- 
ings made  of  the  same  instructor  at  varying  time  intervals. 

Table  XLI  should  be  read  as  follows:    The  mean  differ- 
ences between  tv/o  ratings,  made  2  months  apart,  each  of  46  in- 
structors is  ,51  on  Trait  I.     This  difference,  being  positive,  in- 

^  H.  H.  Remmers,  "The  College  Professor  as  the  Student  Sees  Him," 
Studies  in  Higher  Education,  XI.    Purdue  university  Bulletin, 
Vol.  XXXIX,  No.  6.    Lafayette,  Indiana:    Purdue  university, 
1929.     p. 63. 
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dicates  an  increase  in  scores,    ij'orty-njne  per  cent  of  these  46 
instructors  showed  an  increase  in  the  ratings  received  during 
the  second  term,  while  42  per  cent  showed  a  decrease. 

A  close  study  of  Table  XLI  reveals  the  following: 
(1)  There  is  a  tendency  for  the  scores  to  increase  with  sub- 
sequent ratings.    This  increase  tends  to  become  larger  as  the 
time  intervals  between  ratings  lengthen.     ^2)  The  percentage  of 
teachers  receiving  higher  scores  tends  to  increase  as  the  time 
intervals  lengthen.     (3)  There  is  considerable  variation  among 
the  traits  in  tne  extent  in  which  their  respective  scores  increase 
with  successive  ratings,    in  general  Traits  I,  III,  IV,  IX,  X, 
XI  and  XIII  show  increases  considerably  larger  than  do  Traits  II, 
VII,  XV,  XVI,  XVII.    While  this  increase  in  scores  over  succes- 
sive ratings  cannot  be  attributed,  on  this  evidence  alone,  to 
the  use  of  the  rating  scale,  it  is  significant  to  note  that  those 
traits  on  which  the  instructors  scored  lower  in  the  first  ratings 
are  those  on  which  the  greatest  increases  are  made. 

Another  coinparison  was  made  by  calculating  the  differ- 
ences between  the  first  and  second  ratings  received  by  the  same 
instructors,     in  this  comparison  the  mean  of  all  the  scores  on 
each  trait  received  by  any  given  instructor  from  all  classes 
rating  him  during  any  one  term  was  used.     The  differences  thus 
obtained  v/ere  grouped  together  according  to  the  time  intervals 
between  ratings.     The  same  was  done  for  the  first  and  last 
ratings  given  to  individual  instructors, 

m  Table  XLII  the  increase  in  the  mean  scores  received 
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in  the  second  rating  over  the  first  is  shown  for  each  trait 
and  for  the  average  of  all  the  traits  along  with  the  percentage 
of  instructors  whose  scores  increase  and  decrease. 

In  attempting  to  determine  the  significance  of  the  ob- 
tained increase  in  scores  over  successive  ratings  it  should  be 
kept  in  mind  that  in  general  the  instructors  who  submitted  to 
successive  ratings  were  those  who  received  quite  flattering 
scores  in  the  initial  ratings,    unly  in  the  case  of  relatively- 
very  few  instructors  was  any  coercion  used.    As  a  result  many  who 
received  low  scores  in  the  first  rating  did  not  submit  to  a  se- 
cond one.     It  is  obvious  that  when  one's  initial  ratings  approach 
the  high  end  of  the  scale,  it  is  more  difficult  to  make  any  sub- 
stantial increase  in  subsequent  ratings  than  when  one's  initial 
scores  are  low. 

There  is  a  strong  probability  that  another  factor 
should  be  considered  when  evaluating  the  significance  of  the 
increases  in  scores  in  successive  ratings.    Observations  made 
by  the  investigator  and  by  those  members  of  the  council  on 
Teaching  who  were  most  actively  interested  in  this  project,  led 
to  the  belief  that  students  tended  to  become  more  critical  with 
successive  ratings.     If  this  factor  operated,  the  obtained  in- 
creases are  not  so  large  as  they  ought  to  be  in  order  to  indi- 
cate accurately  the  real  increase  in  teaching  effectiveness,  as 
judged  by  the  students. 

Howe^rer  true  this  last  suggestion  may  be  the  data 
indicate  a  steady  increase  in  the  scores  received  by  the  aver- 
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age  instructor  in  successive  ratings.    This  increase,  while 
not  large,  has  some  significance  as  it  may  be  reasoned  that 
an  increase,  equal  to  the  obtained  mean  increases,  in  the 
rating  of  any  given  instructor,  whose  first  ratings  placed  him 
in,  say,  the  large  middle  group  of  instructor,  would  raise  him 
out  of  this  group  to  the  next  group  above. 


CHAPTER  XI 
SUmiARY  OF  CO^:CLUSIONS 

Ths  ir>vestigation.  repn-r+cQ  in  this  study  was  beijun  in 
the  fall  of  1928  and  continued  until  the  end  of  the  v/inter  cpJiarter, 
1921.  Itjhad  for  its  objective  the  construction  and  testing  of  an 
instrument  for  measuring  the  personal  fitness  and  teaching  ahility 
of  college  instructors,  through  the  medium  of  student  judgment .The 
experimentation  with  this  scale  involved  the  rating  of  703  instnic- 
tore  by  26074  students  in  1359  classes  at  Iowa  State  College.  The 
scale  which  was  developed  is  to  be  known  as  the  College  Teachers 
Rating  Scale, 

THE  PROBLEM 

The  findings  of  the  investigation  were  subjected  to  de- 
tailed statistical  treatment  in  the  attempt  to  find  the  solutions 
to  the  following  problems  regarding  the  scale  and  its  use. 

1.  How  reliable  is  the  College  Teachers  Rating  Scale? 

2.  How  valid  is  this  scale? 

3.  Hovv  large  is  the  spread  of  general  estimate  or  "halo  ef- 
fect" in  the  ratings  made  by  the  use  of  this  scale? 

4.  What  rels.tion  exists  between  the  ability  of  the  student 
and  the  rating  v;hich  he  gives  his  instructor? 

5.  What  relation  exists  bet7;ecn  the  maturity  of  the  student 
and  the  rating  which  he  gives  his  instructor? 

6.  Vkhat  relation  exists  between  the  size  of  the  class  and 
the  average  rating  received  by  its  instmctor? 

7.  Vfhat  is  the  trend  in  the  numerical  values  of  the  scores  re- 
ceived by  individual  instructors  in  successive  ratings? 


-185- 


-186 

CON0LU3IONS 

A  careful  study  of  the.  da.ta  accumulated  in  this  in- 
vestigation enables  one  to  draw  the  following;  conclusionc.  They 
will  \)e  discussed  in  the  same  order  as  the  oroblems,  to  which 
they  refer,  are  listed  a.bove. 

1,  The  obtained  reliability  coefficient  of  .861,008,  to- 
gether with  the  obtained  difference  of  2.01,  between  the  mesxi 
scores  given  by  the  same  classes  to  the  same  instructors  in  two 
ratings  made  two  weeks  apart,  present  satisfectory  evidence  that 
the  College  Teachers  Rating  Scale  possesses  a  rather  high  degree 
of  reliability.    Even  when  based  upon  the  ratings  of  individual 
students  the  obtained  reliability  coefficients  of  .76 ±.014  and 
.57±,013  indicate  that  the  judgment  of  individual  students  con- 
cerning the  quality  of  instruction  to  which  they  are  subjected, 
possesses  considerable  reliability  when  recorded  on  the  scsle. 
However,  the  first-mentioned  coefficient  is  the  m.ost  significant 
one,  since  the  mean  of  the  individual  ratings  given  by  the 
members  of  a  class  would  be  the  value  ordinarily  used  8,s  a 
measure  of  the  teaching  a.bility  and  personal  fitness  of  any 
instructor. 

2,  The  validity  of  the  scale  is  not  so  clearly  derncnstrated 
by  the  data  disclosed  by  this  investigation.    If  one  holds,  as 

do  some  workers  in  this  field,  that  what  the  scale  meaGures  is 
the  reaction  of  the  student  toward  the  instructor  and  his  teach- 
ing, then  the  evidences  of  reliability  of  the  scale  may  be 
regarded  also  as  evidences  of  validity.     Interpreting  the  term 
"validity"  in  this  manner,  it  may  be  held  that  the  College 
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Teachers  Rating  Scale  possesses  a  high  deg;re9  of  validity.  The 
writer  holds  that  this  interoretation  of  the  va,lidity  oerverts 
the  true  meaning  of  the  term.     If  the  scale  is  to  be  regarded  as 
a  measure  of  the  ouality  of  instruction  its  validity  must  be 
measured  by  the  extent  to  which  student  iudg^aent,  exercised 
through  the  medium  of  the  scale,  gives  cj?.  accurate  evaluation 
of  the  instru.ction.     This  is  the  interpretation  of  the  term  which 
was  adopted  in  this  investigation. 

The  evidences  of  this  type  of  validity  for  the  College 
Teachers  Rating  Scale,  which  r/ere  disclosed  by  this  investigation, 
are  not  very  satisfactory.    The  techniques  employed  in  the  study 
of  validity  contained  sources  of  error  difficult  to  control.  The 
data,  show  that,  for  20  classes  of  the  27  classes  involved  in 
this  part  of  the  study,  the  correspondence  between  the  scores 
given  by  the  judges  and  by  the  students  might  be  regarded  as  being 
quite  close.     If  the  scores  given  by  the  Judges  are  taken  as  a 
valid  measure  of  the  quality  of  instruction,  the  mean  of  the 
students'  scores,  in  these  20  classes,  may  also  be  regarded  as 
being  valid.    In  these  classes  the  obtained  m.en  difference  of 
6,58  between  the  raeon  scores  of  the  students  and  those  of  the 
judges  indicates  close  ag-reement  betv?een  the  two.  For  the  remaining 
7  classes  studied  the  obtained  men  difference  of  26,06  indicates 
wide  disparity  between  the  opinion  of  the  students  and  judges. 
The  data  revealed  in  this  investigation  would  indicate  that,  in 
at  least  a  significant  proportion  of  cases,  the  students'  judgment, 
when  expressed  through  the  medium  of  the  College  Teachers  Rating- 
Scale,  would  not  be  a  valid  measure  of  the  teaching  ability  and 
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personal  fitness  of  sji  instnctor,  vjhen  the  basis  of  reference 
is  the  judiTiient  of  experts  exercised  upon  written  reports  of 
observed  lessons,  personality  factors  being  omitted. 

3.  The  obtained  intercorrelation  coefficients  bet^veen  the 
traits,  averaging-  ,47,  would  be  variously  interpreted  by  different 
authorities.     The  obtained  median  of  23.38  between  the  mean  score 
on  the  highest- SCO ring  trait  and  the  mean  score  on  the  lowest- 
scoring  trait  for  each  of  466  classes  indicates  that  the  students 
exercised  considerable  discrimination  as  to  the  extent  to  which 
their  respective  instructors  possessed  the  different  qualities  in 
the  scale.    On  the  strength  of  the  evidence  secured  in  this  in- 
vestigation it  would  seei^  safe  to  conclude  thv^-t  the  "halo  effect", 
while  it  ma3r  operate  to  influence  the  scores  on  the  inclivi'^upl 
traits  on  the  scale,  is  not  so  great  as  to  prevent  the  stud.ents 
from  exercising:  considerable  discrimination  between  the  different 
traits,  when  rating;  their  ins tri:c tors  with  the  College  Teachers 
Rating  Scale. 

4.  The  obtained  coefficient  of  correlation  of  ,15 ±,031 
between  the  grades  given  to  the  students  by  the  instructor,  and 
the  ratings  given  to  the  instructor  by  the  same  students,  make  it 
safe  to  conclude  that  no  significsnt  relationship  exists  between 
the  ability  of  the  student,  as  measured  by  his  academic  grades, 
and  the  rating  which  he  gives  to  his  instructor  on  the  College 
Teachers  Rating  Scale. 

5.  The  maturity  of  the  student  as  indicated  by  the  college 
year  in  which  he  is  classified,  also  appears  to  have  an  insig- 
nificant influence  upon  the  rating  which  he  gives  his  instructor. 
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Ths  obteiaed  ciiff erences,  ranging  from  0.2  to  1.5,  between  the 
mean  scores  given  by  classes  in  the  four  different  college  years, 
seem  email  enough  to  be  disregarded  when  oom-oaring  the  ratinsrs 
received,  by  teachers  from  classes  in  d.ifferent  college  years. 

6.  The  size  of  the  class  does  not  appear  to  have  any  sig- 
nificant inf licence  upon  the  ratings  given  to  instructors. 

Except  in  the  case  of  elapses  containing  less  than  seven  students, 
the  mean  scores  received  b]^  the  saine  instructor  from  classes  of 
varying  sizes  correspond  very  closely.    The  obtained  differences 
between  the  mean  scores  received  by  the  same  instructors  from 
classes  in  the  different  size-groups,  range  from  .07  to  3,43, 
with  a  mean  of  0.67,     This  small  mea.n  difference  indicates  what 
might  r;ell  b*?  regarded  as  very  close  correspondence  betxveen  the 
scores  from  classes  of  different  sizes, 

7.  There  is  a  well-marked  tendency  for  the  scores  received 
by  the  average  instructor  to  increase  with  succeBsive  ratings. 
The  obtained  increases  in  the  mean  scores  received  by  the  same 
instructors,  range  from  1,55  for  a  3-month  interval  between 
ratings  to  S.98  for  the  36-  month  interval.     The  observed  im- 
provement is  quite  gradual,  increasing  as  the  interval  between 
the  ratings  lengthens.     The  percentage  of  instn.ictors  increasing 
their  scores  increases  steadily  from  52  per  cent  in  the  3-month 
interval  between  ratings  to  7S  per  cent  in  the  36-month  interval. 


.CHAPTKR  XII 

SOME  PROBLEMS  FOR  FURTHER  INVESTIGATION. 

Not  a  fev;  problems  related  to  the  rating  of  college 
teachers,  were  encountered  in  this  investi.gs.tion  or  were  sugges- 
ted by  the  experiences  gained  in  the  extended  applico.tion  of  the 
scale  at  lova  State  College.      These  problems  may  be  roughly 
classified  into  two  groups:  A.  Problems  involved  in  establishing 
reliability  and  validity  of  the  rating  of  college  teachers  by 
their  students;  and  B.  Problems,  involved  in  the  meR.surem.ent  pnr= 
improvement  of  instruction  in  colle^^es,  rhich  might  be  investi- 
gated hy  the  use  of  the  rating  scale, 

A.  Problems  Involved  in    Es tablishinp:  the  Reliability 
and  Validity  of  the  Rating  of  College  Te?.ch..rs  by  Their  Students, 

1.  The  discovery  of  a  more  reliable  and  valid  technique 
for  measuring  the  existence  and  extent  of  the  spread  of  general 
estimate,  or  "halo  effect", 

2.  The  effect  of  increasing  the  number  of  items  in  a 
rating  scale  for  evaluating  college  teaching;,  through  the  use  of 
student  judgment , upon  its  reliability  and  validity,  and  the 
determination  of  the  optimum  number  of  items  which  such  a  scale 
should  possess, 

3.  The  determination  of  the  most  essential  abilities  and 
qualities  which  a  college  teacher  should  possess,  by  the  use  of 
more  scientific  method  than  has  heretofore  been  used, 

4.  The  determination  of  the  optimum  number  of  divisions 
or  degrees  of  ability  on  a  scale  for  rating  college  teachers. 

5.  The  effect  rhich  the  omission  of  the  title  for  each 
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trait  in  the  scale  would  have  upon  its  reliobility  and  validity, 

6.  A  more  detciiled  study  of  the  comparative  r'^lisbility 
and  validity  of  the  eeparate  traits  on  the  Colle^-e  Teaching 
Rating  Scale, 

7.  The  determination  of  the  reliability  of  the  judgments 
of  experts  concerning  the  quality  of  teeching  as  reported  by  a 
competent  observer, 

B,  Problems,  Involved  in  the  Measurement  and  Improvement 
of  College  Teaching:,  which  might  be  Investigated  by  the  Use  of 
the  College  Teachers  Rating  Scale, 

1,  The  influence  of  the  extended  and  regular  rating  of 
college  teachers  by  their  students  upon  the  effectiveness  of 
college  instruction.     This  would  involi^e  the  relationships 
between  students,  teachers  and  administrators,  as  T;ell  as  the 
quality  of  the  more  formal  instruction  in  class  room,  and  laboratory, 

2,  A  study  of  the  com.parative  effectiveness  of  instruc- 
tion in  the  d.ifferent  divisions  and  departments  of  the  college, 

3,  The  problem  of  class  size  as  it  affects  the  effec- 
tiveness of  college  instruction  needs  to  be  more  thoroughly 
investigated, 

4,  The  influence  of  the  sex  of  students  and  of  instruc- 
tors upon  the  numerical  value,  the  reliability,  end  the  validity 
of  the  ratings  given  college  teachers  by  their  students. 

5,  The  relationship  between  the  age  and  teaching  ex- 
perience of  college  instmctors  sjid  the  ratings  given  to  them  by 
their  students. 
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6,  The  relation  between  th.;  length  of  acquaintance 
and  number  of  contacts  between  students  and  instructors  and 
the  reliability  and  validity  of  their  ratings. 

7.  The  relation  between  the  mental  ability  of  the 
student,  as  measured  by  mental  tests,  and  the  renting  which 
he  gives  to  his  instructor. 

8,  A  comparison  of  the  ratings  given  by  students  a,nd 
by  heads  of  departments,  or  other  cualified  judges,  to  the  same 
instructors, 

9.  The  relation  between  the  academic  and  professional 
qualif icstions  of  instructors  and  the  ratings  which  they 
receive  from  their  students, 

10,  The  relation  between  the  type  of  subject  taught 
and  the  ratings  received  by  the  instructors  of  each. 

11.  The  modification  of  the  scale  to  make  it  applic- 
able to  the  evaluation  of  leboratorj'-  instruction, 

12,  The  relation  between  the  time  of  day  at  which 
classes  are  held  and  the  ratin.p:  given  to  the  instructors  by 
their  students. 

13.  The  relative  value  of  the  rating  scale  as  a  mesons 
of  approach  to  the  problem  of  the  improvement  of  college  and 
university  teaching.     (See  appendix  B,  Exhibit  3.) 
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IN  THIS  BJVESTIGATION 


■202- 


Print  instructor' 
nauie  here. 


RViTTli^f  SC/VLE  FOP  TEACH-PS 
low",  Strte  Collpr.e. 

Dept . 


Exhibit  1 


Co'jirse  No. 


c^^e^'^llv  :     Please  rnte  this  instructor  on  '-ach  of  the  qualities  bolov:  by 
nakin-  a  check  (  y)  at  that  place  on  each  line  v;hich  r.ost  nearly  corresponds  to 
your  judgment  of  him.     To  illustrate:  Under  "Preoar it  ion  for  Class  Meetin-s",  if 
it  is  your  opinion  that  most  class  mr^etin^s  are  very  carefully  -lanned,  but  that 
occasionally  the  preparation  is  inadequate,  place  check  about  as  shovm  belov;: 


Usually  some  pr^^oar-  Obviously 
ition  but  not  often     littli-  or  no 
adequat'  .  prep^^. ?'-ation . 


PREPAY XTIOK  FOR  CLASS  J'/iEETITiaS 


^   .ii^ 

Evei-y  class 

meeting  very 
careful] y 
planned. 

Do  njt  si'^n  your  name  or  make  any  other  mark  v/hich  mi  :ht  serve  to  identify 
you.     Re-d  over  descriptive  phrases  under  ench  line  carefully  before 
decidinp;  "vvhere  to  place  chock.     Do  nov  confer  v;ith  "nyone:  Use  your  owx 
best  judgment. 


I;^REPAR  :TION  FOR  CL 


II,     BTTEREST    I'^^D  EiITHUSI.\GM 
IN  III.:.  SUBJECT. 


III.  ABI  .I'^Y  Ti) 
IT.T  SUBJECT. 


TV.     FEELING  BITVJEEN  INSTRUCTOR 
AND  STUD'vNTS. 


0?CtANI7ATI0N  of  COURSE. 


1 

!        ;  [ 

1 

hvery  -rla-ss 

Usually  soir.e  prep  r 

-  Obviously 

meeting  very 

;ition  but  not 

often 

little  or  no 

carefully 

ad^^quate . 

preparation. 

plan/^ed. 

Alvfays  ' 

)  in 

:     eems  mildly 

inter 

Subject 

over  ,vith 

sub- 

ested. 

seems  irk- 

ject. 

some  to  aim. 

interest  l 

,  su- 

.itudents  ieem 

iJd- 

■\iority  in- 

illy runs 

hi.rh 

ly  interested. 

attentive 

most  of  period 

Kxc  llont 

Nf. ither  goodvi 

11 

Instructor 

spirit  of 

nor  "'nta^onisn 

tends  to 

"Oodv;ill. 

particularly  i 

n 

antagonize 

evidence . 

class. 

! 

Very  v;ell  or-     .:■  'idently  some  or-    Anparentl  , 
^snized,  ganization  but  not     little  or  no 

complete.  or!::anization, 


n.     THIN  XING  DEJvlAI'IDED  OF 
STUDFT'lTS. 


II.  ABILITy  TO 


J, »»-««««««-«- 

All  rfork  re- 
quires sound, 
accurate  and 
or iginal 
thinking, 
EXPR-'SS  T:w:rHT. .]....  J....  1. 


Thought  •  :\d  mcmor- 
iZ'jtion  used  about 
equally. 


***** 


»«******-*' 


Very  little  or 
no  thinking  de^ 
m:-inded. 


I'll 
**---*******-**i 


Excellent 
choice    'nd  cult  to  grasp  mean- 

arrangement  of  ing. 
•■■ords;  nie-^ning 
always  clear. 


Poor  c -10 ice  ''.nd 

m-ngement  of 
-/.ords;  me  en  ing 
difficult  to 
grasp. 
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711 1.  EN11NC1A.TI0^T. 


IX. 


X. 


XI, 


LE  .Div:2  r-iscussion 


P]:BSOKAL  A.PPEAKVICE. 


XII.     PITE:^"' ST  STUDEIITS. 


XII T.   Sv!:SE  OF  PROPOI'TION. 


XIV.     SINSE  CP  HMMOP. 


-Li.t-«.i  >  »  t  1  t  ]■  t      i  ...  .1.  ...  i  ....]....  L  ....  i  .  1  ...  .1 

>pO' /CT  voi-y      .       1     f..ir;  tv':  tirp.C3      Vr^r  '  poor: 

jlf^t^rly  •u,d        iniictinot  aaid  d::ffi-    -jviords  very 
distimtly.        cult  to  he-  r.  indi.stinot 

and  difficult 
to  he;xr. 

U...i....i....L....l....J....L....I....l....i....| 

Very  shillful      1-air  ability  shoiAir.  Dlscvosior. 
in  directinf^        s.t  tir.ies.  usually  rarr-.b- 

thought.  Dis-  point- 
cussionr  al-  less, 
lA'ays  lively  •~-nd 
pointed. 

k....t....J.«..t«...<....t....l....J....i....l,.,«) 

aJ  ''.iJ'3  veil-  Usi'ally  rather  niovenlyj 

groor.ied;  clotlies  untidy  and  care-        clothes  and 
neat  arid  clean,     less.  person  untidy 

and  ill-kept. 

l....l....i....i....J....J....l....i....i....l....l 

Manner  T^leasing,  Obj ectioiiable  Oonc-ta-itly 

free  from  an-        peculiaritioG  not      exhibits  an- 
r.oyi'-g  manner-      seriou;   or  nu:ner-  noying 
isms.  ous,  mannerisms, 

Xntorerjted  in      Gives  '^.p  riist  'nee  L-Jiintereoted 
students  per-      only  when  request-      in  students 
sonally;  eager    ed,  personally; 
to  help.  unwillin;;-  to 

helvj, 

1  :  !  1  i  i  J  ■>  J  i  t 

Str-usscs  in.por-  uften  stresses  ■'onsist''rnt].;.' 

tant  topics;        trifling  details  emph-' sizes  un- 
,^ives  little        and  slights  impor-  inportart  de- 
attention  to  in-  tant  topics.  tails,  and 
significmt  do-  neglects 
t^ils.  larger  issues, 

..een  S'^r. se  o;'      A.bout  rwerage:  Practicjlly 
humor.  nisTes  many  instpinces  devoid  of 

of  humor,  humor. 


XV. 


SELF- C  CUE  I  DPI!  CE, 


l....L....i....J....i....j...«i.i«i 

Alv;ayG  sure  of    P':-i^-.Ly  se^: -ooi: 

himself;  meets    dent;  occur-ior.ally 
dif  i^iculties  disconcerted, 
vrith  poise. 


hesitant, 
timid,  un- 
certain. 


XVI.     TOLERiuTCE  Ai^D  LIBEHA:L.ITY.   i..^...  j. 


"■'clcomer  diff:-r-  .'Jjiows  occa.'ional 
ences  of  oi^inion,  iiitolerancc . 


Very  in- 
tolerant, 
opinionated. 


Place  this  rating  ir.  envelope  and  seal.     Give  it  to  instructor  st  the  next 
class  meeting.    He  will  fonrard  it  to  the  Council  on  Toacliing  vfithout  opening  it. 
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RATING  SCALE  FOR  TEACHERS 


Exhibit  2 


Print  instructor's  name  here  .  Dep't  Course  No  

Please  rate  your  instructor  as  to  each  of  the  seventeen  qualities  named  below.  * 

It  is  necessary  in  each  case  only  to  place  a  check  mark  ( V )  on  the  accompanying  line  at  tliat  point  which  seems  to  you  to  be 
right. 

The  higher  ratings  are  made  by  placing  the  check  mark  at  the  left  end  of  the  line;  the  lower,  by  placing  it  at  the  right.  These 
stages  and  also  the  middle  ones,  are  clearly  described  by  the  words  printed  just  below  the  line.  The  check  mark  may  in  any  given 
case  be  placed  either  directly  above  the  words  or  along  the  line  between  them  as  you  deem  proper. 

Please  make  this  rating  conscientiously  and  individually.  Your  instructor  will  receive  only  the  final  summary  of  the  results  from 
the  class  as  a  whole.   He  will  use  them  for  self-improvement  in  teaching. 


I.   Preparation  for  class 
meetings 

11.    Interest  and  enthusiasm  in 
his  subject 

ni.  Ability  to  arouse  interest  in 
g  students 

IV.  Organization  of  course 

V.  Scholarship 

VI.  Ability  to  express  thought 

VII.  Enunciation 

VIII.    Thinking  demanded  of 
students 

IX.  Assignments 

X.    Leading  discussion  and 
questioning 

XI.  Sense  of  proportion 

XII.  Feeling  between  instructor 

and  students 

XIII.  Sense  of  humor 
XrV.  Self-confidence 

XV.   Tolerance  and  liberality 

XVI.  Personal  appearance 

XVII.  Personal  peculiarities 


Class  meetings  very  carefully 
planned. 

Usually  some  preparation  ;  often 
seema  inadequate. 

Little  or  no  preparation. 

Very  enthusiastic  and 
interested. 

Seems  only  mildly  interested. 

Subject  seems  irksome  to  him. 

Interest  usually  runs  high. 

Students  seem   only  mildly 
interested. 

IVIajority  inattentive  most  of 
period. 

Course    well-organized ;  parts 
clearly  related. 

Some  organization,  but  not 
always  clear. 

Little  or  no  organization. 

Knowledge  of  subject  broad 
and  accurate. 

Knowledge  apparently  deficient 
at  times. 

Knowledge  very  plainly  defi- 
cient. 

Words  come  easily ;  meaning 
always  clear. 

Some    hesitation    for    words ; 
meaning  at  times  not  clear. 

Much  hesitation  for  words, 
meaning  often  not  clear. 

Speaks  very  clearly  and 
distinctly. 

Words  sometimes  indistinct  and 
not  easy  to  hear. 

Words    very    ilndistiuct ;  often 
impossible  to  hear. 

Work  demands  much  sound, 
original  thinking. 

Thinking  and  memorization 
required  about  equally. 

Thinking    discouraged ;  much 
memorization  demanded. 

Assignments    clear,,  reasuiiable 
and  carefully  given. 

Rather  indefinite  and  often 
hurriedly  given. 

Very  indefinite,  usually  hurried- 
ly given. 

Questions  thought-provoking, 
discussions  lively  and  worth- 
while. 

Questions  usually  call  tor  facts 
or  lead  to  rambling  discussion. 

Few  questions  or  none  put  to 
class. 

stresses    important    topics ; 
disregards  trivial  details. 

Occasionally    stresses  details, 
neglecting  important  topics. 

Often  neglects  important  topics 
for  unimportant  details. 

Feeling  of  good-will  prevails 
strongly. 

Neither  good-will  nor  antagon- 
ism seems  to  prevail. 

Instructor  tends  to  antagonize 
class. 

Has  keen  sense  of  humor. 

Humor  occasionally,  but  not 
often  exhibited. 

Manifests  little  or  no  humor. 

Sure  of  himself ;  meets  difficul- 
ties with  poise. 

Fairly    self-confident ;  occasion- 
ally disconcerted. 

Hesitant,    timid,  uncertain. 

Welcomes  dilterencea  ol  opinion. 

Sometimes  impatient  when  stu- 
dents oppose  his  views. 

Easily  aroused  to  temper  by 
opposition. 

Well-groomed ;  clothes  neat, 
clean,  in  good  taste. 

Usually  rather  untidy  and  care- 
less as  to  appearance. 

Slovenly ;  clothes  and  person 
untidy. 

Manner  pleasing ;  free  from  an- 
noying mannerisms. 

Objectionable  mannerisms  not 
serious  or  numerous. 

Constantly  exhibits  annoying 
mannerisms. 

Please  do  not  sign  your  name  or  make  any  other  mark  which  might  servo  to  identify  you.  Place  this  sheet  in  envelope,  seal  and  give  it  to  instructor  He 
will  forward  it  to  the  council  on  teaohinjr  without  opening  it. 
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Exhibit  3 


lO-^A    STATE  COLLEGE 
Council  on  Teachinfj; 


}Hrch  12,  1929. 


To  Members  of  the  Instrrctin™  St'^ff: 

l^.'e  are  sendin::;  yoi'  n  copv  of  the  rating  scale  whereby 
members        the.' instructin--:  sta'"":"  m^y  learn  the  rea:;tions  of 
students  to  their  tea  chin?';. 

Its  use  may  be  requested  by  the  department  head;  or  if 
this  is  not  done,  a  tc-^.cher  may  still  use  it  voluntarily  as  a 
means  of  improving  his  vv-ork.     In  the  latter  case,  the  results 
of  the  rating  will  not  be  nade  Icno-.vTi  to  uiiy  of  his  superiors. 

If  the  scales  are  used  in  the  maviiier  last  suggested,  they 
are  given  to  the  students  in  unsealed  envelopes  at  any  meeting 
in  the  last  week.     The  last  fifteen  minutes  should  be  set  nside 
for  it.     After  giving  the  proper  instructions,  the  instructor 
will  asl:  that  the  shopts,  vfhen  filled  out,  be  placed  in  the 
hands  of  a  designated  student,  who  will  later  brin'r;  them  to 
Kim,     Then  the  instructor  should  probably  leave  the  room  in 
order  that  there  be  no  uneasiness  on  the  part  of  students  lest 
he  be  observing  how  he  is  being  marked. 

Tt  is  probably  not  advisable  for  any  instructor  to  have^ 
the  ratings  made  by  more  than  tvra  classes  unless  the  latter 
are  small.     This  sug":estion  is  made  because  the  machinery  for 
handling  the  returns  has  not  yet  been  tried  out  on  a  large 
scale  and  there  is  some  danger  that  it  be  overtaxed. 

If  you  -.vish  to  ma>.e  use  of  the  plan,  --.dll  you  ]:indly  tell 
us  the  number  of  rating  scales  that  you  ;7ill  need.  Early 
notTce  wiTrTnsure  their  arrival  in  time  together  vith  in-- 
structions  for  their  use. 

Sincerely  yours, 

COU^iCIL  on  TEACHING, 

By       H.  Lancelot, 

Chairman. 
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DIRECTIONS  TO  BE  GIVEN  TO  STUDENTS  MEN  RATING  IS  MDE. 


Note:     The  last  fifteen  to  ei£;hteen  minutes  of  a  class  period 
should  be  set  apart  for  the  rating. 

In  order  that  the  results  be  comparable,  it  is  necessary  that 
all  r-^tings  be  made  in  the  same  manner  with  the  same  general  in- 
structions given  to  all  classes.     The  procedure  should  be  about  as 
follows : 


After  the  envelopes  containing  the  rating  scales  have  been  dis- 
tributed and  the  sheets  removed,  the  instructor  should  say: 

"It  helps  us  greatly  in  our  teaching  if  our  students  will  give 
us  their  candid  impressions  of  our  work,  I  am  asking  that  you  give 
me  yours;  and  in  order  that  you  may  be  candid,  it  has  been  arranged 
that  I  shall  not  see  personally  any  of  these  sheets  which  are  to  be 
used  in  rating  me,  after  you  have  filled  them  out,  but  instead  only 
the  general  summary  of  the  opinions  of  the  class  as  a  whole. 

"Will  you  first  read  carefully  the  instructions  at  the  ton  of 
the  sheet;  and  then  will  you  note  as  carefully  the  words  belovif  the 
line  in  each  case  before  you  decide  just  where  the  check  mark  should 
be  placed  upon  the  line? 

"I  shall  leave  the  room  while  you  are  making  the  ratinr:,  which 
s-hould  require  about  ten  minutes,  '"hen  you  have  finished,  will  you 
place  the  sheet  in  the  envelope  _  again,  seal  it,  and  give  it  to  f/[r,  ■ 
(or  Miss)  ,  who  will  deliver  them  to  me?" 


(instructor  then  loaves  room,    ''Vhen  the  envelopes  are  brought 
to  him  by  the  student  designated,  he  should  tie  them  in  a  bundle 
and  send  them  by  campus  mail  to  the  Council  on  Teaching,  318  Agri- 
cultural Hall,     The  results  will  be  reported  as  soon  as  they  can 
be  tabulated,  which  will  probnbly  be  v/ithin  two  weeks.) 
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REPORT  OF  RATING  BY  CLASS 


Exhibit  4 


Instructor 

Quarter 


Course 


Year 


IIo.  in  class 


TRAITS 


Distribution  of  grades  given  by  students 


90--:  80-  :  70-  :  60-  :  50-  :  40- :  30-  :20-:10-:  0- 
100:89   :79   :69   :59   :49   :39   :29   :19   :  9 


Average 


I.  Preparation  for  class  : 
meetinfrs  : 

■  

:  

II.  Interest  a  d  enthusiasi 
in  his  subject 







III.  Ability  to  arousp  in- 
terest in  students 



■ 

•  



IV.  Org-T.ization  of  course 

':  

V.  Scholarship 

-  ..  - 





1  — 

•-  

VI.  Ability  to  express 
thought 

[   



 ' 









•  ' 



VII.  Enunciation 







-  ■  ~ 

-  —  

VIII,     Thinkinr:  denanded 
of  students 





IX.  As  :;i gnr.ien ts 



_  





S.  Leading  discussion 
and  questioning 

XI.  Sense  of  proportion 

XII.  Feelin,^  between  in- 
structor and  students 

XIII.  Sense  of  humor  : 

XIV.  Self-confidence  : 

XV.  Tolerance  and  liberality 

XVI,  Personal  appearance  : 

XVII.  Personal  peer liarities 

COimCIL  ON  TEACHING 


APPENDIX  B 
INSTRUCTIONS  TO  JUDGES  MD  SPECIMEN 
REPORTS  OF  LESSONS  OBSERVED 

PRACTICAL  USES  FOR  A  COLLEGE 
TEACHERS  RATING  SCALE 
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Exhibit  1 

SUGG-ESTIONS  FOR  EVALUATING  THE  QUALITY  OF  INSTRUCTION 
DESCRIBED  IN  THE  RECORDED  OBSERVATIONS 

Attached  to  the  records  of  each  instructor  is  a  copy 
of  the  rating  scale  which  it  is  the  purpose  of  this  study  to 
standardize.    The  last  six  of  the  qualities  listed  on  it  are 
personal  traits.    We  have  indicated  by  a  check  mark  at  each  of 
these  traits  the  rating  given  to  the  instructor  by  the  students 
in  the  class  observed.     It  was  thought  that  this  would  aid  you 
in  evaluating  the  quality  of  the  instruction  described  in  the 
recorded  observations. 

We  are  asking  you  to  indicate,  in  the  same  manner, 
your  opinion  of  the  instruction  described,  by  checking  nine 
of  the  first  eleven  items  on  the  scale.    The  items  which,  it 
seems,  would  be  impossible  to  evaluate  from  the  records  and 
which  therefore  will  not  be  checked  are  "scholarship"  and  "enun- 
ciation."   I  believe  that  the  remainder  can  be  estimated  from 
a  reading  of  the  observations. 

The  number  of  students  in  the  class  and  the  year  of 
their  course  in  which  the  subject  is  offered  are  indicated 
at  the  top  of  the  rating  scale.    Our  suggestions  are  that 
you  study  carefully  the  rating  scale,  especially  that  part 
of  it  which  you  are  to  check,  and  get  well  in  mind  the  various 
items  to  be  considered  in  arriving  at  your  estimates  of  the 
quality  of  the  instruction.    The  records  should  then  be  care- 
fully read  and  as  good  a  picture  of  the  actual  work  in  the  class 
room,  as  is  possible,  obtained  from  them.    You  will  then  be 
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ready  to  record  your  estimates  on  the  scale  as  indicated 
above.    Use  should  be  made  of  the  statements  under  the  lines 
when  you  are  recording  your  opinions. 

When  you  have  completed  the  work  would  you  be  good 
enough  to  bundle  everything  together  and  send  the  package  to 
me  by  express  collect.    Do  not  neglect  to  enclose  your  bill 
for  the  service  rendered. 

I  assure  you  that  your  cooperation  is  greatly  appre 

ciated. 
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Exhibit 

Instructor;  1  First  lesson  observed 

Course:  English  May  20,  1929 

The  instructor  opened  tVie  class  'if,ith  a  humorous  re- 
mark regs.rding  Veishea,   the  annual  all-college  celebration 
which  had  been  staged  the  last  three  days  of  the  preceding 
week . 

Instructor:  "Nov/,  in  connection  with  this  story  we 
are  reading  today,  vshat  is  the  first  thing  which  had  been 
brought  out,   setting  or  characterization?" 

Student:  "The  setting." 

Instructor:  **';ihat  is  your  opinion  of  the  one  which 
should  come  first?" 

Student:  "In  this  story  tVie  setting  should  come  first 

Instructor:  "Does  t^^is  apply  to  all  stories?  (Stu- 
dents agreed  it  did  not.) 

"What  place  has  tVie  setting  in  a  story?" 

Student:  "It  is  the  foundation." 

TViis  idea  was  enlarged  upon  to  some  extent. 

Instructor:  "Kow  well  do  the  setting  and  the  char- 
acterization in  this  story  fit  together? 

"Could  you  take  the  sanje  ch-racter  and  set  him  down  i 
Des  Moines,  Paris,  London,  or  any  other  large  city? 

"In  how  large  a  town  can  a  person  dominate  and  be  a 
characteristic  of  the  place? 

"How  large  a  town  was  this  one?    V/as  it  small  enough 
to  allow  one  person  to  be  a  character*?" 
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The  above  questions  were  answered  by  students  quite 
satisfactorily.     The  ides  seemed  to  be  one  of  simply  guiding 
the  thought  of  the  class. 

Instructor:     "I  am  going  to  read  something,  which, 
after  I  am  through,  I  v»ant  you  to  tell  me  if  it  fits  our 
captain." 

He  proceeded  to  read  a  description  of  a  sv.asVibuckling 
*hard -boiled*  soldier  of  fortune. 

After  this  he  went  ahead  to  describe  this  captain  in 
the  story,  as  he  conceived  him,  \',ith  such  questions  as  tVie 
following: 

•'Is  he  likely  to  be  a  garrulous  chap,  a  noisy  talka- 
tive fellow? 

**WfLere  do  you  find  this  fellow  most  often?" 
Student:  "In  a  cafe." 

Instructor:   "\Vbat  is  a  cafe?    A  cafe  in  France  would 
have  what  chara.cteris tics?" 

Several  students  attempted  to  describe  the  interior  of 
a  French  cafe. 

Instructor:  "I  want  a  verb  to  describe  how  this  man 
ate  his  meals." 

The  instructor  followed  this  up  and  endeavored  to 
stimulate  the  imagination  of  the  students  and  to  lead  them  to 
interpret  the  story  and  get  a.  mental  vision  of  the  interior  of 
the  cafe  and  of  the  characters  in  the  story. 

Instructor:  Now,  why  would  the  captain  select  such  a 
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cafe? 

'♦Would  you  select  it,  J--? 

"I  w&nt  you  to  see  th^^t  the  captain's  condition 
physically  was  very  satisfactory.     He  had  b  good  pension, 
good  health  and  enough  to  provide  for  all  his  physical  wants. 
It  was  up  here."   (Drawing  a  line  on  the  "board  near  the  top) 

Instructor:  "Now  how  much  does  a  man's  physical  con- 
dition determine  his  happiness*?" 

The  students  agreed  th^^t  for  some  men  the  physical 
condition  would  iDe  a  very  large  factor. 

The  instructor  next  drew  a  verba.l  picture  of  the 
history  of  the  American  people  from  the  pioneer  sta.ge  to  the 
present  one  of  affluence  and  then  asked: 

"Do  you  think  the  present  generation  is  any  happier 
than  the  older  one?" 

Some  discussion  by  students  followed  this  question. 

Instructor:  "'ATiat  is  the  cause  of  most  people's  wor- 
ries?" 

The  idea  cf  financial  difficulties  was  brought  out 
here  and  ela.borated  upon. 

Instructor:  "Physica.lly  our  captain  was  at  the  top. 
'iThere  was  he  spiritually?  (Ke  drew  a  line  at  the  bottom  of  the 
board.)     What  had  he  ever  done  for  humanity?" 

"What  happened  to  change  his  spiritual  life?" 

Student:     "Ee  got  acquainted  with  a  little  girl." 

Instructor:     "Let  us  draw  a  picture  of  this  little 
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girl." 

This  the  instructor  proceeded,  to  do,  sided  "by  sug- 
gestions from  students  in  response  to  his  questions. 

A  line  was  drawn  st  the  "bottom  of  the  blackboard  to 
represent  the  physical  condition  of  this  poor  little  girl. 

The  point  was  also  made  th^t  there  is  something  in 
human  nature  vjViich  responds  to  the  good. 

Instructors     "V/hy  did  the  coptsin  wsnt  to  help  this 
little  girl?" 

Different  opinions  were  volunteered. 

Instructor:     "Y/hen  he  took  her  to  live  v^ith  him, 
whr^t  happened  to  his  physical  condition?" 

The  line  representing  his  physical  condition  was 
moved  down  to  the  center  of  the  blackboard. 

Instructor:     "What  happened  to  him  spiritually?" 

The  line  representing  the  spiritual  condition  was 
moved  up  to  the  top. 

Instructor:     "Now  this  is  to  illustrate  a  great  prin- 
ciple in  life  and  one  which  appears  in  a  great  many  stories. 

"Is  there  a  verse  in  the  Bible  about  this?     'Ke  that 
loses  his  life  shall  find  it.*     Service  does  not  give  a.  man 
beautiful  clothes,  fine  houses,  etc.     but  it  brings  peace  and 
happiness . " 

Instructor:     "For  Wednesday  I  want  you  to  write  in 
class  on  any  question  with  regard  to  characterization  and  set- 
ting--"how  to  see  a  story,  how  to  present  a  moral  problem. 


-214- 

'•The  story  vie  shall  h«ve  for  Friday  is   »Vis  et  Vir'" 

N.B,     A  good  many  of  the  questions  v»ere  of  the 
elliptical  type  ending  in  v.hat,  why,  etc.     The  teacher  also 
has  3  h«bit  of  repeating  students*  answers. 
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Instructorj     1  Second  lesson  observed 

Course:    English  Kay  24,  1929. 

Instructor:  "Cur  assignment  for  next  day  is  Virginia 
end  I  viant  ycu  to  study  this  story  carefully  for  characteriza- 
tion and  setting.     This  morning  Vve  have  the  story  of  Vis  and 
Vir.     V/hat  lesson  was  the  author  trying  to  teach  in  this  story? 
What  principle  seems  to  lie  behind  this  story?" 

Student:  "Carelessness  and  bravery." 

Instructor:  "What  is  his  theme  regarding  carelessness, 

C  ?'♦ 

First  Student:  "Carelessness  causes  trouble." 

Second  Student:  "Carelessness  must  be  punished." 

Instructor:  "Vi/hat  about  the  bravery?" 

Student:  "Bravery  will  be  rewarded." 

Instructor:  "Do  you  think  that  this  is  true  to  life? 
Is  carelessness  always  punished  and  bravery  rewarded?" 

There  was  some  discussion  of  the  trutVi  of  this  in 
actual  life. 

Instructor:  "You  like  the  story  but  you  don't  think 
it  was  effective,    V/hat  stood  in  the  way  of  its  effectiveness?" 

The  student  endeavored  to  sViow  that  it  was  very  im- 
probable. 

Instructor:     "A  story  must  be  true  to  life.     Is  human 
nature  as  portrayed  in  this  story?"     (This  question  was  put  to 
several  students.) 

Instructor:     "Let  us  look  at  another  phase.    Was  this 
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a  time  of  pesce?" 

Student:  "No,   it  v/as  at  the  time  cf  the  French 
Revolution . 

Instructor:   "How/  many  of  you  know  anything  of  the 
French  Revolution?" 

One  student  gave  a  very  hrief  description. 

Instructor:  "V/hat  v»ere  the  key  \\ords  of  the  French 
Revolution?" 

After  scrae  unsuccessful  attempts  hy  several  of  the 
students  the  instructor  supplied  the  words — Liberty,  Frater- 
nity and  3quality. 

Instructor:     "Let  us  see  v,hs  t  is  the  difference  he- 
tvieen  those  times  and  ours,     7/ould  we  at  this  time  kill  a  man 
"because  of  carelessness  in  performance  of  duty?" 

This  was  discussed  and  some  of  the  students  produced 
evidence  to  show  that  men  were  c our tmartialed  and  shot  in  the 
last  war  for  failing  to  carry  out  orders. 

Instructor:  "Here  is  a  man  who  was  careless  at  one 
time  and  very  brave  at  the  next;  he  was  decorated  with  a  medal 
at  one  moment  and  the  next  he  was  shot.     Does  this  seem  right 
to  you?    V/hy  was  it  done?" 

Different  ideas  were  volunteered  by  students. 

Instructor  (After  graphically  describing  the  scene): 
"Wiat  is  your  emotional  response  to  such  an  act?  Why  did  he 
not  take  advantage  of  the  opportunity  to  escape?" 

This  provoked  quite  a  spirited  discussion  among  the 
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students,  a  considerable  proportion  of  the  class  taking 
part . 

Instructor:  "Let  us  come  to  it.     Do  you  think  that 
the  effect  created  by  his  punishment  vvoula  he  fa.vorahle?" 

Instructor:  "How  many  kings  are  there  today?" 
(Several  were  named) 

Instructor:  "V/hy  are  there  so  few  now?" 

Student:  "The  people  hsve  become  educated  to  believe 
that  no  one  man  ha.s  the  power  to  rule  over  the  lives  of 
others . " 

Instructor:  "That  is  a  good  statement.    We  don't 
think  that  now, but  what  is  our  principle  now  concerning  the 
right  of  a.  person  to  control  others?" 

The  idea  of  representative  government,  the  responsi- 
bility to  do  the  right  thing  not  because  it  is  ordered  from 
above  but  because  it  is  the  right  thing  to  do  for  the  wel- 
fare of  all  was  emphasized  here. 

Instructor:  •^WitVi  this  principle  in  mind,  let  us 
examine  our  story  again  and  inquire  as  to  its  fairness.  How 
many  thought  that  this  was  fair  treatment?" 

The  majority  thought  that  it  was  not  quite  fair  and 
in  the  discussion  reference  was  made  to  the  V/orld  V/ar  and  to 
the  higher  regard  in  which  human  life  is  held  now  than  in  the 
time  of  Napoleon, 

Instructor:  "What  has  brought  about  this  change?  Is 
it  an  attribute  of  democracy?'* 
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Here  he  referred  to  fraternity  life  Vwith  its  hazing 
and  initiation  stunts  and  ordeals  vihich  ere  supposed  to  in- 
spire loyalty  and  respect  for  the  organization,  and  raised 
the  question  as  to  hovs  effectively  this  was  accomplished  by 
such  means.    He  followed  this  up  "by  a  description  of  rough 
methods  of  some  athletic  coaches  who  by  abuse  and  rough 
treatment  endeavored  to  inspire  Viis  men  to  play  hard.  His 
opinion  was  that  this  was  absolutely  the  wrong  method  to  use 
in  getting  the  best  out  of  men. 

Instructor:  "You  have  your  assignment  for  next  day. 
Do  the  best  you  can  vv  ith  it." 
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Instructor:  1 
Course:  English 


Third  lesson  ohserved. 
May  27,  1929 


Instructor:  "V/ith  regard  to  your  stories,  I  would 
like  you  to  come  to  ray  office  as  soon  as  you  can  and  vie 
will  go  over  yours  together 

"Now,  we  have  for  today  the  story  of  Eugenie  and 
Old  Grsndet.    '#ho  are  the  characters  in  this  story?" 


These  were  named  by  the  students. 

Instructor:  "V/hy  was  Old  Grande t  well-to-do?" 

First  Student:  "He  had  a  good  trade." 

Second  Student:  "He  made  good  investments." 

The  instructor  remarked  that  many  of  us  would  he 


able  to  make  good  investments  if  we  Viad  anything  to  invest. 

Instructor:  "7/as  this  old  man  the  same  kind  of  man 
as  old  Ma.tre  Hauchenau  in  the  Piece  of  String?" 


Most  of  the  class  thought  that  they  were  alike. 
Instructor:  "V/hat  was  the  indication  of  miserliness 


Student:  "He  picked  up  a  piece  of  string," 

The  consequences  of  this  innocent  act  were  reviewed 


Instructor:  "Can  a  msn  exist  long  without  the  re- 


The  force  of  public  opinion  was  brought  out  at  this 


Instructor:  "Now,  what  is  the  difference  in  the  case 


in  the  case  of  Ma.tre  Hauchenau?" 


briefly . 


spect  of  his  fellows?    V/hy  can  he  not?" 


point. 
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of  old  Grandet?" 

The  point  that  he  v!,as  o  real  miser  with  a  love  for 
gold  vvas  brought  out. 

Instructor:  "Do  you  see  any  differences  in  the 
chars.cters  in  this  story?     Is  there  a,ny  figure  who  could 
he  contrasted  with  Cld  Grandet? 

Student:  "Charles*   father,  Old  Grandet brother 
was  quite  different.     He  had  failed  in  business," 

Instructor;  "What  made  him  go  bankrupt?" 

Student:  "He  was  not  a  good  investor," 

Instructor:  "V/hy  was  he  not  a  good  investor*?" 

Student:  "He  thought  too  much  of  Charles,** 

The  instructor  here  drew  a  word  picture  of  the 
typical  spender  who  cuts  a  wide  swath  and  leaves  his  family 
penniless. 

Instructor:  "Now,  what  is  the  purpose  of  this  story?" 

Student:  "To  show  the  difference  between  two  ex- 
tremes--miserliness  and  wasteful  spending." 

Instructor:   "'t/lThich  one  of  these  two  gets  the  more 
criticism? " 

It  was  generally  agreed  that  the  miser  was  criticized 

more  • 

Instructor:  "V/hich  is  more  valuable  to  society?"  I 
suppose  you  all  think  that  Old  Grandet  was  a  pretty  hard  old 
tightwad 

First  Student:  "I  thought  that  he  was  too  mean  and 
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too  hard  on  Viis  daughter." 

Second  Student:  "Maybe  she  needed  a  strict  hand." 

A  short  discussion  followed  regarding  the  influence 
of  being  strict  viith  children.     From  this  the  instructor 
described  the  tendency  which  exists  in  our  day  to  live  fast 
and  spend  more  than  we  earn.     To  make  this  more  real  he 
used  the  Kentucky  Derby,  which  had  recently  been  run. 

Instructor:  "Which  one  of  these  characters  would 
have  been  there?    Would  they  have  betted?" 

At  this  point  the  instructor  gave  description  of 
personal  acquaintances  who  were  of  the  type  of  Charles 
Grandet . 

Instructor:  "Have  we  this  type  today?" 

The  class  were  certain  that  such  men  existed  today 
and  reference  was  made  to  the  situation  at  a  sister  university 
concerning  the  athletic  department  there.     The  point  was  made 
that  the  trouble  was  caused  by  alumni  who  are  of  the  nature 
of  Charles  Grandet. 

Instructor:  "Is  it  time  for  us  to  be  thinking  which 
one  of  these  plans  you  will  follow?    I  despise  the  miser  but 
I  also  despise  the  spendthrift,  the  man  who  spends  everything 
for  social  display  and  leaves  his  family  penniless.  Where 
are  you  going  to  make  your  path?" 

Student:  "I  am  going  to  try  to  take  a  middle  course." 

Instructor:  **Y/ha.t  is  the  middle  course?" 

Here  followed  some  problems  and  remarks  concerning 
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the  present  high  costs  of  living,  the  struggle  to  maintain 
social  standards  and  various  other  aspects  of  economic  life. 

.  Instructor:  "How  should  a  man's  income  relate  to 
his  expenditures?" 

Student:  "A  man  should  save  ten  per  cent  of  Viis 

salary, " 

Instructor:  "V/ould  this  be  enough?" 

This  led  to  a  brief  calculation  of  the  amount  that 
this  would  be  for  the  average  family  and  it  was  decided  that 
10000  dollars  would  cover  it. 

Instructor:  "If  a  man  on  a  salary  of  2000  dollars 
saved  this  nuch,  what  would  his  wife  and  children  have  said 
about  him?" 

Instructor:  "There  is  another  angle  to  this.  liThat 
change  came  over  Old  Grandet?    Did  he  really  love  his  daugh- 
ter and  want  her  to  be  happy? 

Instructor:  "Did  Charles'  fa.ther  change?    Which  child 
would  make  the  greatest  contribution  to  the  world?" 

The  instructor  brought  the  discussion  to  a  close  by 
a  rather  spirited  defence  of  the  old  Purita.n  virtues  of 
simplicity  and  thrift  and  expressed  the  wish  that  the  rural 
middle  west  would  quit  aping  Uew  York  City  and  other  sophis- 
ticated sections  of  the  country  and  hold  to  the  simpler  life 
of  the  founders  of  the  country. 

Instructor:  "Your  next  assignment  is  the  two  stories- 
ITrancois  Villon  and  A  Lodging  for  the  Night." 
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In  struct  or:  .1 
Course:  Englisti 


Fourth  lesson  observed. 
Msy  29,  1929 


The  instructor  passed  out  news -letter ,  sent  cut  by 
Sigma  Upsilon  honorary  literary  society. 

He  next  showed  how  he  liked  the  final  papers  to  be 
put  up,  running  over  some  specimens  he  hsd  brought  in  with 
him. 


class  which  are  nearly  good  enough  to  enter  our  Ames  Narra- 
tives.    I  propose  to  change  our  program  and  instead  of  tak- 
ing up  Frajacois  Villon,  we  will  take  up  a  few  of  these 
stories . " 


teresting  stor^^  involving  college  life,  commenting  on  the 
finer  points.     The  reading  consumed  about  thirty  minutes 
of  the  period. 


Instructor;  •*!  have  about  five  stories  from  this 


With  this  corraiient  he  proceeded  to  read  a  very  in- 


Instructor:  "Well,  what  about  it?" 


First  Student:  "Pretty  good," 


Second  Student:  "I  enjoyed  it  better  than  anything 


I  have 


read  in  a  book." 


Instructor:  "\7hy  did  you  enjoy  it?" 


First  Student:  "Eecause  the  scenes  are  familiar  to 


us ." 


Second  Student:  "I  don't  believe  we  have  characters 


like  these  around  our  campus. 


This  provoked  conflicting  opinions  from  several  in 
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class.     The  instructor  agreed  that  although  the  characters 
were  not  typical  it  vta.s  perfectly  all  right  to  use  them. 

Instructor;   "What  other  criticisms  have  you?" 

Student:  "I  don't  believe  that  gold  could  he  picked 
up  like  tViat." 

It  was  agreed  that  gold  did  often  occur  as  nuggets. 

Student:  "I  don't  believe  that  Indians  would  have 
done  this  (kill  white  people)  as  late  as  fifteen  years  ago." 

This  was  instantly  disputed  by  several,  who  gave 
instances  of  rather  recent  uprisings  in  the  South  and  V/est 
against  the  'i/hites. 

Instructor:  "Have  any  others  any  criticism?     I  have 
some  other  good  stories,  but  time  will  permit  only  the  read- 
ing of  a  brief  book  report," 

Of  this  he  had  time  to  read  only  a,  small  portion  of 
one  before  period  was  over. 

Instructor:  "Go  to  the  library  and  read  one  of 
Kipling's,  Stevenson's  or  Poe's  short  stories  and  give  a.  re- 
view of  it.     Write  a  synopsis,  state  what  the  conflict  is, 
who  the  characters  are,  what  are  the  dramatic  events  and 
the  climax.     Tiake  full  use  of  me  in  conference  over  your 
stories . " 
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Instructor:  2  First  lesson  observed. 

Course:    Engineering  Kay  14,  1929 

The  class  observed  was  a  sophomore  class  of  nine 
students , 

The  instructor  opened  the  class  by  making  a  joking 
remark  about  the  Green  Gander,  college  humorous  publication 
vshich  several  students  were  reading. 

Instructor:  "I  am  anxious  to  discover  whether  you 
understand  thoroughly  wha  t  v.  e  were  dealing  with  last  day. 
Take  a  piece  of  paper  and  give  tVie  different  methods  of 
selecting  the  elements  of  any  figure  when  you  are  finding 
the  coordinates  of  the  centroid  by  the  method  of  integration. 
You  will  have  about   six  minutes." 

At  the  end  of  five  minutes,  one  minute  warning  was 
given,  at  tVie  end  of  which  the  papers  were  collected. 

Instructor:  "How  many  are  sure  you  have  them  right?'* 

Only  three  in  the  class  were  sure.     The  instructor 
went  to  the  board  and  drew  the  following  sketcVies: 


Instructor:  "How  are  you  going  to  select  the 
elem.ents  required?" 

A  quite  spirited  discussion  followed,   into  which 
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practically  all  students  entered  voluntarily.     After  this 
the  instructor  summarized  the  conclusion. 

Instructor:  "Is  it  clear  now  to  those  who  were  not 

sure?" 

Reference  was  made  to  the  text  where  topic  is  dis- 
cussed . 

Instructor:  "Have  you  any  questions  to  ask  about 
the  exarrjples  given  in  the  text?" 

Student:  "I  don't  see  how  he  gets  rid  of  the  r^  in 
his  equation," 

The  instructor  took  the  text  and  followed  the  equa- 
tion through  step  by  step,  attempting  to  explain  it  to  the 
student.     This  failing  he  went  to  the  board  and  explained 
it  by  sketch  and  equation. 

Instructor:  "Any  more  questions?    How  many  can  take 
any  of  these  examples  and  work  them  out  without  any  trouble? 
How  would  we  do  this  one?" 

The  instructor  drew  the  following  sketch  on  the  black- 
board: 


Instructor:  "How  can  we  locate  centroid  of  the  area 
between  the  "Y"  axis  and  the  parabola? 

"V/hat  is  our  general  equation,  K  ?" 
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He  worked  out  the  equation  step  "by  step,  having 
difficulties  cleared  up  either  "by  himself  or  by  students. 
The  class  displayed  considerable  interest  "by  voluntarily 
entering  into  the  discussion,  challenging  one  a.nother*s 
reraarks  and  contributing  their  own  ideas. 

Instructor:  "Kow  let  us  try  another  one.     Pass  to 
the  blackboard,     I  shall  have  you  all  take  the  same  one-- 
Find  the  center  of  the  lateral  surface  of  a  right  circular 
cone . " 

The  students  went  to  the  board  and  worked  on  the 
problem  but  the  end  of  the  period  came  before  they  had  all 
finished.    The  class  had  to  be  dismissed.     The  instructor 
told  them  they  would  find  this  problem  in  the  text. 

Instructor:  **Por  next  time  take  Problem  195.  You 
will  need  this  in  your  course  in  materials  next  fall." 
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Instructor:  2 
Course:  Engineering 


Second  lesson  observed 
Kay  16,  1929 


Instructor:  "Some  time  ago  I  mentioned  the  use  of 
centroids  in  the  strength  of  materials.    Kow  many  understand 
their  use?" 

Several   said  they  did  not  and  the  instructor  ex- 
plained it  hy  sketch  and  by  reference  to  painters  working 
outside  of  the  building  at  the  time  standing  on  a  plank 
supported  at  each  end.    A  new  term,  "^moment  of  inertia  of 
deflection"  was  introduced.     The  instructor  told  the  class 
they  would  run  across  this  in  a  week  or  two. 

Instructor:  "Another  use  we  have  for  this  is  in  the 
design  of  fly-wheels,  I  am  going  to  ask  you  to  do  this  for 
Tuesday." 

Here  he  drew  the  following  sketch  on  the  blackboard: 


T 

1 


T" 


1% 


dia. 


Instructor:  "I  wa.nt  you  to  find  the  weight  of  the 
rim,  assuming  that  the  flywheel  is  made  of  cast  iron. 

The  assignment  was  made  clear  by  answering  the 
questions  asked  by  the  students. 

Instructor:  "Are  there  any  questions  about  the 
lesson  for  today?"   (There  were  none). 

"Then,  I  would  like  to  give  you  this  problem  to  work 


i 
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in  class.     Suppose  you  had  a  pole  thirty  feet  long,  fourteen 
inches  in  diameter  at  hutt  and  six  inches  at  top  and  V4e  had 
to  load  it  on  two  trucks  so  that  there  is  equal  load  on  each 
truck.     If  one  truck  is  placed  four  feet  from  large  end  how 
far  from  small  end  should  the  second  truck  be  placed?"  (He 
drew  a  sketch  of  r)ole  on  the  blackhoard  with  dimensions.) 

"Let  us  see  how  good  your  judgment  is,     V/here  do 
you  think  the  second  truck  will  he?" 

Estimates  made  hy  cla.ss  ranged  from  nine  to  sixteen 
feet  and  were  placed  on  the  blackboard. 

The  class  was  then  sent  to  the  blackboard  to  work 
it  out.     It  wss  observed  that  some  students  copied  the  work 
of  others.     This  was  a  short  period,   thirty  instea.d  of  the 
usual  fifty  minutes  and  the  class  hsd  to  be  dismissed  before 
any  one  had  completed  the  problem. 

Instructor  (Before  dismissing  class):  **How  many 
think  you  can  take  the  centroid  of  the  area  of  the  cross 
section  and  arrive  at  correct  answers?    How  many  think  you 
have  to  find  the  mass  and  volume?  (Class  was  divided  on 
this).     Better  think  over  this  and  we  will  finish  it  next 
time  .** 
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Instructor:  2  Third  lesson  observed 

Course:  'Engineering  May  21,  1929 

The  instructor  opened  class  "by  collecting  papers. 

Instructor:  "Last  day  Y^e  were  working  on  a  problem 
and  the  question  arose  ss  to  vihether  we  should  use  the  area 
or  volume  in  arriving  at  its  correct  solution,     Kov/  many 
think  you  could  use  the  area?     (Only  one  did)     Hotv  many 
think  it  would  be  necessary  to  use  the  volume?     (All  the 
others  did). 

Instructor:  (To  first  student)  '*How  would  you  solve 
it  by  using  the  area?" 

The  student  attempted  to  outline  his  procedure,  the 
instructor  following  him  with  sketches  and  equations  on  the 
board. 

Other  members  in  the  class  v.ere  eager  to  challenge 
his  method  and  to  point  out  where  the  error  was  and  why  it 
would  not  work.    The  instructor  encouraged  them  to  do  so, 
forcing  them  back  on  their  ground  by  challenging  their  propo- 
sitions and  criticisms.    Evidently  he  was  attempting  to  help 
the  weaker  side,  in  this  case,  the  first  student  mentioned. 
Several  students  went  to  the  blackboard  to  explain  or  demon- 
strate their  proposals,  one  going  up  without  being  asked  and 
while  others  were  discussing  a.nother  proposed  solution,  and 
drawing  sketches  to  demonstrate  his  method.  Considerable 
interest  was  shown,  several  students  wishing  to  speak  a.t  once 
and  all  being  included  in  discussion. 
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After  the  class  hsd  exhausted  all  their  ideas  and 
the  answer  still  remained  doubtful,  the  instructor  by 
questioning  led  thera  through  the  reasoning  step  by  step 
and  showed  how  working  from  the  area  would  not  give  cor- 
rect solution. 

Instructor:  "ITovs ,  what  is  the  point  to  all  this? 
Why  did  we  spend  so  much  time  on  this?** 

Student:  "So  thst  we  would  never  make  the  mistake 
of  taking  the  area  instead  of  the  voliiiae.'* 

Instructor:  "Yes,  after  this  you  should  remember 
that  in  finding  centroids  it  is  best  not  to  try  too  many 
short  cuts," 

He  now  gave  the  correct  answer  to  the  problem  and 
reminded  them  of  the  tentative  solutions  they  had  submitted 
last  day  when  the  problem  was  first  presented  to  them. 

Instructor:  "I  would  like  now  to  point  out  our  next 
work.    The  lesson  runs  from  page  180  over  a  few  pages  to 
end  of  chapter.     There  are  a.  few  important  things,  I  want  to 
point  out,  which  you  should  learn  and  be  able  to  reproduce 
at  any  time.     (These  were  enumerated), 

"The  review  scheduled  for  Saturday  will  include  to- 
day* s  lesson. 

•♦Now  for  our  problem  for  today--How  much  did  this 
fly  wheel  weigh?" 

This  question  was  asked  of  all  members  and  weights 
ranging  from  338  to  5100  pounds  were  reported,  the  majority 
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"being  around  5000. 


The  small  one  (338)  was  soon  explained  and  ex- 


planation caused  some  merriment  at  the  expense  of  the  stu- 
dent making  the  error. 

The  instructor  asked  for  difficulties  met  vjith  and 
these  were  explained. 

Instructor:  "I  think  we  can  get  started  on  a  nevu 
prohlem, " 

Here  he  drsv/  the  follovving  sketch  of  steam  engine 
governor  with  "brief  explanation  of  same: 


lems  you  will  have  problems  involving  governors.    V/hat  is 
the  centroid  of  the  rod  and  "ball,   on  this  governor,  v<ith 
respect  to  its  **Y"  axis? 

**How  would  you  go  a"bout  solving  this  problem?** 
Some  steps  were  suggested  which  were  briefly  de- 
scribed on  the  blackboard. 

Instructor:  **In  the  last  problem  we  used  volume. 
Can  we  do  it  this  time?    If  we  put  the  "X"  axis  through 
the  top  of  rod  would  the  result  be  the  same?" 


Cent  iron 


Instructor:     "In  your  mechanical  engineering  prob- 
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Instructor:  2  Fourth  lesson  olDserved. 

Course:  Engineering  May  25,  1929 

Instructor:  "Let  us  see  v\hst  we  knovv  or  don't  knovj 
about  the   'moment  of  inertia,'     Gould  an  area  have  a  moment 
of  inertia?" 

Student:  "No." 

Instructor:  "Why  not?" 

It  "wsas  brought  out  the  t  the  proper  term  for  this 
function  'flas  "second  moment,'* 

Instructor:  "What  do  you  understand  by  'the  polar 
moment  of  inertia?"* 

A  student  attempted  an  answer  which  was  followed 
up  by  tVie  instructor  through  questioning  this   student  and 
others  who  were  gradually  brought  into  the  discussion.  The 
following  are  typical  questions  asked  by  the  instructor: 

"Gould  a  moment  of  inertia  be  negative? 

"Is  the  polar  moment  always  greater  than  the  other 

two?** 

By  such  questions  and  by  sketches  on  the  blackboard 
the  instructor  led  students  to  think  through  the  various 
aspects  of  the  topic  of  moment  of  inertia. 

Instructor:  "What  is  the  radius  of  gyration  Mr,  S--?" 

To  make  this  question  clearer  the  instructor  drew  a 
sketch  on  the  board  and  restated  the  question  in  terms  of  the 
concrete  example  represented  by  the  sketch.    The  first  stu- 


dent  did  not  give  a  satisfactory  ansvier  and  another 
attempted  to  give  a  better  one  in  vshich  he  suggested  that 
the  radius  of  gyration  around  a  centroidal  axis  would  be 
zero , 

Instructor:  "Howi  many  think  that  it  ought  to  be 
zero  around  a  centroidal  axis?" 

Only  one  thought  so  and  the  instructor  showed  by 
substituting  actual  values  in  the  formula,  that  this  would 
not  be  possible. 

One  student  held  that  this  radius  of  gyration  was 
an  imaginary  quantity  but  it  was  necessary  in  order  to  solve 
certain  problems. 

Instructor:  "Y/hat  sort  of  problems?" 

The  class  could  not  a.nswer  so  the  instructor  told 
tViem  that  in  working  with  loads  on  columns  this  function 
was  used. 

Instructor:  "Wha.t  a.bout  the  parallel  axis  and  the 
moment  of  inertia?" 

This  was  cleared  up  satisfactorily  by  the  students. 

Instructor:     "^You  seem  to  have  the  theory  down 
pretty  well.     The  next  step  is  to  use  it  in  solving  problems," 

First  Student:  "There  are  a  couple  of  examples  in 
the  text  which  I  don't  understand,"     (Here  he  gave  numbers 
of  the  exajnples). 

The  instructor  took  the  class  through  these  examples 
by  questioning  elucidated  by  sketches  on  the  blackboard. 
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Instructor:  "\7hat  wss  your  question  Mr.  A--?" 

Student:  "The  same  one--I  understand  it  noiv," 

Instructor:  "Is  there  any  other  question?" 

One  student  submitted  one,  "which  "was  cleared  up 
in  the  sa.me  manner  a,s  described  above.     In  this,  as  in  the 
others,  the  instructor  did  not  lecture  but  rather  directed 
the  thinking  of  the  students  through  the  problems  by  question- 
ing and  illustrations. 

Instructor:  **Let  us  see  if  vv  e  can  apply  these  prin- 
ciples . 

The  students  were  sent  to  the  blackboard  and  numbered 
in  two's;  the  ones  were  given  one  problem  and  the  two's  a- 
nother  and  different  one.    They  proceeded  to  work  independ- 
ently on  the  problems,  the  instructor  aiding  individuals  by 
questions  and  suggestions  only  when  called  upon.     The  students 
seemed  quite  interested. 

Instructor:  "Let  us  take  our  seats  for  a  minute. 
L^r ,  J--  will  you  go  to  the  blackboard  and  explain  your  so- 
lution of  the  problem." 

This  was  done  to  satisfaction  of  all.     The  instruct- 
or then  asked  ajnother  student  to  explain  his  solution  of 
the  second  problem.    After  this  was  done  the  instructor 
announced  a  written  review  next  time  and  dismissed  the  class. 
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Iristructor:  3  First  lesson  observed 

Course:  Horticulture  iiovembsr  IS,  1920 

Mote:     This  class  consists  of  about  forty  sophomore  students 
classified  in  the  Division  of  Agriculture. 

The  instructor  opened  the  class  meeting  by  asking  a 
student  to  make  a  draiwing  of  the  cross  section  of  a  tree  on 
the  blackboard.     To  another  student  he  assigned  the  task  of 
dravsing  a  plat  of  an  orchard,    V/hile  this  was  being  done  he 
passed  back  to  the  class  examination  books,  with  the  follow- 
ing remarks:     "These  answers  were  unusually  good  but  there 
are  a  few  things  which  might  be  said  before  we  leave  them. 
The  first  question  concerns  the  cambium  layer.     This  is  of 
the  greatest  importance.    Mr.  C--  made  a  perfect  grade  on 
this  question  and  we  sha.ll  ask  him  to  explain  it."  (The 
student  began  his  explanation  and  the  instructor  interjected 
the  following  questions):    "How  old  is  the  twig  in  your  sketch?" 
(Student  sa.id  it  was  two  years  old).     "Have  you  the  width  of 
cambium  in  the  right  proportion  to  the  rest  of  the  twig?" 

This  question  was  passed  around  and  it  was  finally 
decided  that  the  cambium  should  be  shown  nearer  the  outside 
of  the  tree. 

Instructor:     "ITow,  what  is  the  role  of  the  cambium?" 
Student:     "The  cambium  is  the  growing  layer  of  the 

tree . " 

Instructor:     "A  good  many  of  you  said  that  it  was  a 
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storehouse  for  food.  Could  it  be  used  for  carrying  the  food 
up  and  down?" 

Student:     "I  don't  believe  that  it  could." 

Instructor:     "This  reminds  us  that  Curtis  of  Colum- 
bia University  has  the  theory  that  the  cajnbium  does  carry 
food  but  it  is  not  generally  believed." 

Here  the  instructor  showed  by  arrows  on  the  board, 
the  direction  of  the  movement  of  the  sap  up  and  dov.n  the 
tree,  with  the  cambium  between.     One  student  asked  the 
question  as  to  how  the  sap  got  into  the  tree  from  the  b^rk. 
The  instructor  explained  by  referring  to  the  medullary  rays 
in  the  oak  woodwork  of  the  room,  mentioning  the  term "quar- 
tered" in  this  connection. 

Student:     "Does  the  stuff  which  comes  on  a  bruise 
on  a  cherry  tree  come  from  the  cambium?'* 

Instructor:     "I  am  not  sure  that  it  does.     V/e  must 
look  this  up.    Let  us  turn  to  the  next  question--new  tissue 
formation.     Do  you  remember  how  the  author  treated  this?  He 
compared  it  to  tVie  making  of  waffles.     In  the  making  of  waf- 
fles you  need  what?" 

The  instructor  with  the  help  of  the  only  girl  stu- 
dents in  the  class  enumerated  the  ingredients  of  waffles. 

Instructor:     "If  you  had  only  half  as  much  soda  as 
needed  and  still  wanted  good  waffles  what  would  you  do?" 

Student:     "Tighten  up  my  belt." 

Instructor:     "V/hich  are  the  ones  which  limit  growth- 
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the  ones  in  smaller  or  larger  amounts?"     (Much  dispute  in 
class  about  this  question,  and  the  instructor  explained  hovi 
the  absence  of  tVie  very  small  amounts  of  iron  and  sulphur 
Vihich  are  required  makes  plant  grov;th  impossible).     "In  our 
country  here  which  one  limits  growth?     Is  it  within  the  pov;er 
of  the  grovser  to  control  growth?" 

Student:     "You  cannot  control  the  water  contents" 

Instructor:     "Vraatl    You  can't  I    Y/hy  was  Mr,  P  

ploi/aing  under  that  rye  in  his  orchard  the  other  day?" 

Another  student:     "It  would  increase  tVie  humus  con- 
tent and  in  this  way  increase  the  holding  power  of  the  soil." 
(V/ith  some  little  comment  the  next  question  was  taken  up). 

Instructor:     "An  excessive  production  of  leaves  in- 
dicates what?    Here  we  have  two  pear  trees,  one  grafted  on 
quince  and  one  on  pear.    Which  would  have  the  larger  quan- 
tity of  leaves?" 

Student:     "The  standard  tree,  the  one  grafted  upon 
the  pear,"     (This  was  accepted  and  instructor  elaborated  up- 
on i  t ) . 

Instructor:     "The  sixth  question  was  answered  cor- 
rectly by  everyone."     (Reviewed  it  briefly,  with  sketch  on 
board).     "l^ow  for  the  next  one?"    (Used  orchard  plat  put  on 
the  board  by  the  student  at  the  first  of  the  hour,  illustrat- 
ing the  conditions  of  the  question  by  labelling  the  differ- 
ent parts  of  the  sketch). 

Instructor:     "In  this  section  the  trees  would  be 
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growing  too  fast.     V/ould  you  prune  them  to  stop  their  growth?" 
Student:     "Yes,  I  would  prune  it," 

Instructor:     "Here  (referring  to  sketch)  v<e  sre  not 
pruning  very  close  and  we  get  what  kind  of  growth?"  (fol- 
lowed out  the  other  three  sections  in  the  sketch  in  much  the 
same  manner,  asking  many  questions  ending  in  'what?*.  Some 
students  asked  questions  voluntarily). 

Instructor:     "V/hat  will  be  the  result  of  pruning 
like  this?"     (Showing  a  twig  in  which  the  ends  have  ceased 
to  make  any  shoot  growth  but  have  turned  into  spurs). 

Student:     "You  will  soon  not  have  enough  shoot 

growth , " 

Instructor:     ""AHaat  will  "be  the  result  of  this?  V/ill 
the  plant  bear  every  year?" 

Student:     "lie,  once  every  two  years." 

Instructor:     "If  a  tree  grows  vvell  but  does  not  have 
any  blossoms,  what  is  your  trouble?" 

Student:     "There  is  probably  too  much  nitrogen," 

Instructor:     "Is  it  a  matter  of  nutrition  or  of 
temperature?"     (Here  instructor  wrote  on  board:     Topic  Six-- 
(retting  seed  and  fruit  from  blossoms),     "Here  is  a  patch  of 
strawberries,  which  bloom  and  bloom  but  they  never  form  ber- 
ries." 

Student:     "The  flowers  have  not  been  pollinated." 
Instructor:     "But  suppose  that  you  were  called  on  to 
diagnose  the  case,  what  would  you  do?"     (Here  the  instructor 
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took  some  flower  blossoms  vjhich  he  had  brought  in  and  hold- 
ing them  close  to  the  students  in  the  front  rov.  tore  them  up, 
asking  the  students  to  discover  why  the  flower  had  formed  no 
seed.     He  placed  the  stigma  of  the  flower  on  his  arm  so  all 
could  see  it  and  questioned  student  as  to  the  significance 
of  its  absence  or  presence.     Then  he  placed  the  following  on 
the  board  under  the  heading  referred  to  above: 

1.  Causes  of  failure  of  blossoms  to  set, 
(a)     Imperfect  flowers. 

Instructor:     "I  have  a  patch  of  Dunlap  strawberries 
a.nd  they  set  a  good  deal  of  fruit.     If  you  had  looked  at 
the  flowers  what  would  you  have  found?" 

Student:  "Stamens," 

Instructor:     "Yes,  and  if  we  had  a  patch  of  Dunlaps 
near  a.nother  patch  which  has  no  stamens,  but  both  patches 
set  fruit--whst  has  the  Dunlaps  done?" 

Student:     "Fertilized  the  others," 

Instructor:     "Read  up  in  your  text  on  this  matter 
of  fertilization  and  sterility.     It  is  often  met  with  in 
practical  work," 

Glass  dismissed. 
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Instructor:  3  Second  lesson  observed 

Course:  Horticulture  November  20,  1930 

'iflHiile  the  last  students  vsere  coming  in  the  in- 
structor took  up  the  question  asked  st  the  lest  meeting  re- 
garding the  gum  vfViich  exudes  from  a  bruise  in  the  bark  of  a 
cherry  tree.     This  he  explained  to  them  to  their  satisfac- 
tion, after  vshich  he  called  the  roll. 

Instructor:     "I  would  like  to  put  it  up  to  a  vote 
of  the  class  what  you  would  do  if  you  had  some  very  choice 
apples  on  exhibit  and  a  student  took  one  and  ate  it.  This 
Tisas  doen  recently  by  a  member  of  this  class."  (instructor 
related  how  one  of  the  boys  had  played  a  .joke  on  him  by 
bringing  an  apple  to  the  laboratory  where  the  others  were  on 
display  and  by  making  a  show  of  eating  it  surreptitiously. 
The  class  enjoyed  a  good  Isugh). 

Instructor:     "Unf ortuna. tely  we  shall  have  to  close 
up  Topic  6.     Wciat  is  the  primary  object  of  the  blossom  in 
the  wild  plant?" 

Student:     "To  provide  seed," 

Instructor:     "On  the  whole  what  will  give  you  the 
more  vigorous  growth--cros3  bred  or  inbred  plants?" 
Student:     "A  standard  bred.*' 

Instructor:     "V/hat  do  you  mean  by  standard  bred?" 
(This  was  settled).     "would  seed  from  a  plant  which  ferti- 
lized itself  or  from  a  plant  which  is  fertilized  by  other 
pla.nts  be  more  vigorous?"     "\7hat  did  Darwin  say  about  this? 
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'  Na  tur  e  abho r  s '    h&  t  T" 

Student:  "Inbreeding;," 

Instructor:     "Yes,  but  that  is  putting  it  rather 
strongly  perhaps.     Tones,   of   ,  has  a  differ- 
ent theory,     (Explaining-  it).     To  r^hct  lengths  vsill  nature 
go  to  do  this  cross  breeding?     (Passed  around  blossoms  of 
8  flower},     ifhevQ  is  stigma  located?" 

Student:     "On  the  inside  of  the  corolla," 

Instructor:     "An  insect  lighting  on  this  flower 
would  have  to  pass  into  this  corolla  and  as  he  goes  in  does 
he  touch  the  stigma?"     (By  similar  questions  the  instructor 
showed  how  insects  carry  pollen  and  help  fertilize  plants). 

Instructor:     "The  supreme  object  of  the  flower  in 
the  wild  is  to  produce  what?" 

Student:  "Seed." 

.Instructor:     "Is  that  the  object  of  the  floricultur- 

ist?** 

Student:  "Yes." 

Instructor:     "Do  you  think  so?" 

Student:     "Without  seed  we  cannot  propagate." 

Instructor:     "Do  you  really  believe  that*?"  (Student 
admitted  his  error),     "Is  it  possible  to  get  fruit  without 
having  seed  in  it?"   (Several  examples  were  mentioned). 

Instructor:     "Suppose  we  had  a  block  of  seedless 
oranges  and  there  were  a  lot  of  seedlings  around  about  them. 
■i'Paat  would  you  do?"     (Results  cf  cross  fertilization  from 
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these  seedlings  were  pointed  out).     "Kov;  do  plants  develop 
fruit  without  forming  a  seed?" 

The  instructor  explained  briefly  the  process  and 
told  an  interesting  story  of  hovs  a  woman  during  the  c  ivil 
war  had  "brought  up  some  scions  from  South  America,  which 
scions  after  many  years  were  found  to  produce  seedless  oranges. 

Instructor:     "In  nature  what  was  the  function  of 
the  flesh  of  the  fruit?" 

Various  students  suggested  several  purposes  and  the 
instructor  pointed  out  how  nature's  purpose  had  been  reversed 
by  man  and  now  the  flesh  is  the  important  object  in  seed  pro- 
duction. 

Instructor:  (To  the  only  girl  in  class).  "Did  you 
ever  see  a  strawberry  blossom  lacking  a  pistil?" 

While  student  was  answering  the  instructor  started 
an  outline  on  the  board  by  writing: 

V/hy  fruit  fails  to  set-- 

(a)  Flowers  lacking  stamens  (imperfect) 

Instructor:     "All  over  the  country  the  Bartlett 
pear  is  a  poor  setter.     \7hy?    Has  it  a  perfect  flower?  Yes. 
>7/hat  is  the  trouble?" 

(b)  Flowers  perfect  but  self-sterile. 

Student:  "Why  don't  they  set  fruit?"  (instructor 
passed  the  question  around). 

Instructor:     "As  a  plant  becomes  weakened  by  neglect 
it  first  becomes  self-sterile  and  later  inter-sterile." 
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The  students  asked  several  questions  concerning  the 
application  of  this  phenomenon  to  practical  problems  of  fruit 
growing  and  the  instructor  explained  hy  using  concrete  il- 
lustrations from  different  rjsrts  of  the  country.     The  follow- 
ing was  written  on  the  boa.rd  below  the  past  point: 

(c)  Flowers  are  perfect  but  inter-sterile. 
Instructor:     "Suppose  I  had  a  patch  of  I'uscctin 

grapes  which  never  furnished  seed.     If  you  exajnine  them  you 
will  find  stamens  but  they  are  defective." 

(d)  The  stamens  are  often  defective. 
Instructor:     "Did  you  ever  see  potato  seed?"  (Few 

had  and  the  instructor  explained  why  they  very  rarely  devel- 
oped in  Iowa),     "V/ould  the  opposite  be  true?     Did  you  ever 
see  a  wild  grape  blossom?    Y/hat  is  defective  there?" 

(e)  The  pistils  are  defective. 

(f)  Male  and  fen:als  flowers  on  separate  plants. 
This  point  evoked  considerable  discussion,  several 

students  entering  freely  into  the  conversation  and  showing 
more  than  a  little  interest.     The  instructor  proceeded  to 
open  up  a  new  list  on  the  board,  v.riting  the  title  c.  s  fol- 
lows: 

Conditions  Outside  Plant-- 

Instructor:     "V/hen  I  was  a  student,  ;7alter  F  was 

working  on  trie  fig  tree.  They  were  not  bearing  and  he  went 
to  .Europe  to  study  it.  They  told  him  that  nothing  could  be 
done.     He  afterwards  found  that  a  certain  small  fig -like  plant 


harbored  a  'Aasp  which  could  fertilize  the  fig  tree,  etc." 

Upon  calling  upon  the  students  for  other  conditions 
the  following  were  enumerated  snd  written  on  the  board: 

{&)  Disease  may  destroy  blossom, 

(b)  V/eather  conditions  unfavore'ole 

1.  Frost 

2.  Cold  and  r&iny  weather 
5.  Lust 

(c)  Lack  of  insects  to  pollinate 
(d^  Strong  and  untimely'"  sprays 
(e)  Lack  of  vigor  in  the  tree 

vVhile  placing  the  above  on  the  board  the  instruct- 
or illustrated  each  by  giving  examples. 
Class  dismissed. 
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Instructor:  3  Third  lesson  observed 

Course:  Horticulture  November  25,  1930 

The  instructor  opened  the  class  meeting  "by  mention- 
ing the  beauty  of  the  campus,  as  it  had  snowed  during  the 
nigVit  and  snow  vvas  still  in  the  trees.    He  observed  that 
this  campus  v.ss  counted  among  the  three  most  beautiful  ones 
in  the  v«hole  country. 

Instructor:     "Before  beginning  I  lAOuld  like  to  re- 
fer you  to  the  sermon  preached  by  Dr.  Paul  Hutchinson  la.st 

Sunday,    Dr.  Hutchinson  painted  a  gloomy  picture.    An 

article  by  Glenn  Frank  in  the  last  issue  of  the  Century  con- 
tains much  the  same  idea.     Both  of  these  men  mentioned  the 
change  taking  place  in  industrial  orga.niza  ti  on ,  ^.hereby  the 
old  relations  betvseen  the  employer  ajnd  the  employee  have 

broken  dovin.   The  question  has  arisen  also  in  connection 

v<ith  horticulture,  for  some  believe  that  the  proper  tViing  to 
do  is  to  connect  orcharding  and  all  sort  of  horticultural 
production  on  a  large  scale  with  immense  acreages  while  others 
believe  that  the  small  farm  is  better."     (About  13  minutes 
spent  thus)  "How,  let  us  go  ba.ck  to  our  problem." 

Here  the  instructor  v.rote  on  the  blackboard  the 

following   sentences,  which  were  the  beginning  of  an  outline 

which  was  developed  as  the  lesson  progressed. 

Development  of  Fruit  in  Ripening 
Things  that  hinder  d evelopment-- 
1.  Physiological  causes. 

(a)  Winter  frsezing--f lower  buds,  branches, 
trunks,  roots. 
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The  following  discussion  took  place  as  this  out- 
line was  "built  up: 

Instructor:     "Wha.t  is  the  greatest  single  factor 
under  this  head?" 

Student:     "The  chanf;e  in  the  season." 

Instructor:     "This  is  pretty  vague.     V/ha  t  would  you 
say,  Mr.  B--?" 

Student:     "The  winter," 

Instructor:     "Yes.     V/inter  injury  to  what  part?" 
Student:     "The  roots." 

Instructor:     "Do  you  think  so?     I  expect  the  roots 
would  he  the  last." 

Another  Student:     "The  shoots." 
Instructor:     "ITo,  I  don't  "believe  so." 
Another  Student:     "The  leaf  hud." 

Instructor:     "How  could  you  tell  which  were  the  leaf 
buds  and  whether  they  were  injured?"     (Gave  instances  of 
government  distributing  foreign  plants  to  individuals  with 
the  request  that  they  report  how  they  stood  the  winters,  etc.) 

The  students  gave  several  answers  to  the  above 
question.     These  answers  were  challenged  and  checked  up  on 
until  the  proi)er  one  car'e  out.     The  class  is  made  up  of  stu- 
dents from  several  different  departments  in  agriculture  and 
the   instructor  constantly  made  application  of  the  facts  dis- 
cussed to  the  different  phases  of  agriculture  represented, 
mentioning  the  students  names  and  the  department  in  which 
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they  v«ere  classified.     At  the  end  of  tViis  p&rt  cf  the  dis- 
cussion the  instructor  wrote  the  follov\,ing  upon  the  board  — 
2.  Conditions  vthich  fsvor  winter  injury. 

Instructor:     "In  the  literature  whst  is  prohably 
the  greatest  cause  cf  winter  injury  mentioned?"  (Several 
students  suggested  different  causes  but  the  instructor  ac- 
cepted only  one  of  these:    Lack  of  maturity.     This  was 
written  on  the  board). 

Instructor:     "Y/liat  are  some  of  the  causes  of  im- 
maturity?" 

Student:     "It  might  be  too  dry." 

Instructor:     '*Yes,  drought  followed  by  fa.vorable 
growing  conditions."     (V/rote  on  board) 

Student:  "Late  cultivation."  (This  also  was  writ- 
ten on  the  board,  after  the  instructor  had  remarked  that  in 
some  cases,  late  cultivation  is  jxracticed  to  deter  blossom- 
ing in  order  to  escape  the  early  frosts) . 

Instructor:     "Another  cause?" 

Student:     "In  places  on  the  tops  of  hills  where  the 
snow  blows  off  the  trees  usually  suffer."     (Instructor  wrote 
on  board:    Lack  of  ground  protection). 

Student:     (After  the   instructor  had  developed  the 
last  point  somewhat)     "I  should  think  a  crop  about  one  foot 
high,  which  would  stand  up,  would  be  better  than  rye  which 
lies  down," 

Instructor:     "-vVhat  you  say  is  true  and  we  will  run 
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across  it  in  another  topic." 

The  instructor  placed  upon  the  hoard. - 
(3)  V/ays  of  avoiding  vi inter  injury. 

Instructor:     '*V/hy  can  the  Europea,n  countries  v<hich 
are  much  farther  north  than  we  are  grow  fruits  which  we  • 
have  difficulty  with?" 

Student:     "The  climate  is  milder," 

Second  student:     "Cultivation  for  centuries." 

The  instructor  accepted  these  answers  and  v.  rote 
upon  the  board: 

(a)  Selecting  favored  site 

(b)  Selecting  hardy  varieties. 
Instructor:     "How  are  you  going  to  protect  them  in 

winters  like  this?" 

(c)  Covering  low-growing  plants. 
Instructor:     "Does  color  make  any  difference?  LTr. 

C--,  between  green  and  purple  which  would  you  select?" 
Student:     "Purple. •* 
Second  student:     "The  green." 
Instructor:     "Why  would  you  take  the  purple?" 
Student:     '^The  dark  would  absorb  the  heat  during  the 

day." 

Second  student:     "But  the  green  would  not  suffer  so 

badly." 

The  instructor  described  the  results  of  some  experi- 
ment which  had  been  csrried  on  by  means  of  putting  thermome- 
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ters  in  twigs  of  different  colors.  The  following  points 
viere  tVien  added  to  tViose  already  on  the  "board: 

(d)  Color  of  twig 

(e)  Spraying  with  lime  sulphur 
Instructor:     "For  next  day  make  a  note  of  how  to 

treat  winter  injured  trees." 
Class  dismissed. 
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Instructor:  3  Fourth  lesson  observed 

Course:     Horticulture  December  2,  1920 

Instructor:     "Just  s.  "word  stout  the  assigned  read- 
ing.    Has  anyone  taken  H  's  book  on  Horticulture  from  the 

table  in  the  office?     It  has  disappeared  and  that  is  against 
the  rule."     (This  matter  was  cleared  up).     "The  assigned 
readings  have  been  for  the  purpose  of  giving  you  what  some 
psychologist  calls  an  apperceptive  mass,  s  basis  on  vvhich 
to  base  our  discussions,  so  they  should  have  been  read  in 
advance  of  the  classroom  discussion.     I  am  afraid  that  some 
of  you  are  not  doing  so  and  in  the  future  I  must  insist  on 
your  doing  this  part  of  the  v;ork.     There  a.re  not  many  rea,d- 
ings  on  our  next  topic.     I  am  passing  out  blank  cards  on 
which  I  wish  you  to  write  the  answers  to  three  questions  I 
shall  give  you.'* 

The  following  questions  were  written  on  the  board: 

1.  V/hat  do  you  mean  by  a  fingicide  and  why  is  it 

effective? 

2.  What  do  you  understand  by  the  word  insecticide 
and  into  what  two  classes  do  we  group  insecticides? 

3.  If  you  spray  for  each  separate  ill  to  which 
horticultural  plants  in  Iowa  are  heir  you  will  make  thirty 
spraying  applications.     How  would  you  avoid  doing  this"? 

The  students  asked  for  explanations  of  the  questions, 
either  because  they  could  not  see  them  or  because  their  mean- 
ing was  not  clear.     The  instructor  was  forced  to  explain  their 
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meaning.  At  the  end  of  about  five  minutes  he  collected 
the  papers.  While  students  v;ere  v;riting  the  instructor 
placed  upon  the  board  the  following: 

Topic  Seven — ^Development  of  ?ruit  and  Ripening. 
1.  Factors  that  hinder  development. 

a.  Physiological  injuries  vshich  are  the  causes 
for  about  one-third  of  loss. 

Instructor:     "Nov^  on  Thursday  v,e  shall  have  to  cover 
picking,  grading  and  packing.    Let  us  see  if  v;e  can  today 
cover  the  period  up  to  the  time  of  ripening,     V/e  have  already 
taken  up  some  of  this.     I  vssnt  to  refer  you  again  to  the 
bulletin  from  Minnesota,  entitled  "V/hy  is  a  Cull  a  Cull?" 
The  writer,  Gaston,  is  perhaps  the  best  authority  we  have 
on  the  marketing  of  fruit.     He  is  now  beginning  a  study  for 
his  doctor's  degree,  wheeling  a  push  cart  on  the  streets  of 
Detroit,"     (The  instructor  cited  a  few  other  instances  of 
the  ways  in  which  this  man  gained  Viis  information). 

Instructor:     "V/e  have  covered  the  different  physical 
causes.     Has  anyone  a  different  angle  to  suggest?"  (in- 
structor wrote  the  following  on  board). 

b.  Fungus  diseases  and  insects 

1.  Fungus  diseases  not  controlled  by  spray-- 
Fire  Blight,  Cedar  Rust. 

The  examples  cited  above  were  not  written  on  the 
board  until  the  instructor  had  related  an  experience  he  had 
on  a,  western  ranch  while  a  young  man.     The  farmer  for  whom  he 


-253- 


worked  had  discovered  what  seemed  to  be  frost  injury  in  the 
morning  after  a  mild  nigVit. 

Instructor:     "»Vhat  do  you  suppose  iwas  the  cause, 
¥oc ,  V/. — ?"     (student  did  not  know). 

Another  student:     "Fire  blight." 

Instructor:  "What  kind  of  s  disease  is  Fire  Blight?" 
Student:     "A  fungus  disease.** 

Instructor:    Let  us  come  over  nov;  to  our  questions. 
What  is  a  fungicide?" 

Student:     "SometViing  which  kills  fungi." 

Instructor:     "V/hy  is  it  effective?" 

This  topic  was  follovsed  through,  bringing  out  the 
ways  in  which  fungi  take  in  their  food  and  how  the  fungicide 
destroys  them.     Q,uestions  and  analogies  were  freely  used  by 
the  instructor.     After  this  he  wrote  on  board: 

2,  Those  that  cs.n  be  destroyed  by  spray. 

Instructor:     "G--,  how  did  you  answer  this?" 

Student:     "I  did  not  answer  it?" 

Instructor:     "Peters,  what  did  you  say?" 

Student:     "By  smothering  or  burning." 

Another  student:     "The  spray  forms  a  coating  over 
the  leaf." 

Instructor:     "And  what  are  the  substances  which  are 
toxic  to  these  fungi?" 

Student:     "Lime  is  one." 

Instructor:     "Is  that  what  the  lime  is  used  for?" 
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"The  tvio  substances  are  copper  and  sulphur,  but  if  esch  is 
applied  separately  v^hat  wiill  they  do?" 

The  follo\iving  were  added  to  outline  on  the  board: 
Copper  and  Sulphur, 

Copper  sulphate  and  lime  3-3-50  (Bordeaux) 
Sulphur  and  lime  boiled  together  (Lime  sulphur) 
Instructor:     "llov*  \\e  come  to  insects  that  cannot  be 
controlled  by  spraying.     Are  there  any  that  cannot  be  con- 
trolled?" 

Student:     "The  San  Jose  Scale," 

Instructor:     "I  don't  tViink  v«e  can  be  sure  about 

that." 

Another  student:     "The  Buffalo  Tree  Hopper," 
Another  student:     "Cinch  Bugs."   (This  last  insect 
M!5S  emphasized  and  described  as  some  of  the  students  indi- 
cated that  they  did  not  knov*  it). 

Instructor:     "Then  V(e  have  the  fourth  group--in- 
sects  v;hich  can  be  controlled  by  spraying,     YPast  two  groups 
ha.ve  v.e  of  these  insects?"     (This  v.as  easily  answered  and 
the  instructor  did  not  dwell  on  this  classification),  "Let 
us  take  the  other  factors.     There  are  three  stages  in  the 
development  of  fruit  which  we  must  recognize: 

1,  The  fruit  set--the  first  three  weeks — all  cell 
division  takes  place  tVien, 

2.  StarcVi  formation  period--from  about  first  of 
June  to  first  of  September, 
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Instructor:     "VAist  color  is  the  fruit  then?" 
Student:  "Greenish," 

Instructor:     "Then  v^hat  can  it  do  itself?"     It  ca.n 
develop  and  will  it  take  in  oxygen  and  give  out  carbon- 
dioxide?" 

Students:  "Yes." 

Instructor:     "ITow,  don't  you  see  the  reason  v^hy  vse 
apply  the  nitrogen  only  at  certain  times?" 

3.  The  period  of  maturation  or  maturity  which  is 
three  or  four  "weeks, 

"ITow  we  will  jump  over  the  questions  of  soil  manage- 
ment.    It  is  so  important  that  I  hate  to  risss  it  over." 

Student  (interrupting):     "'^That  is  the  answer  to  the 
last  question?" 

Instructor:     "ViHiy,  we  simply  mix  the  spray." 

The  instructor  started  to  give  the  assignment  for 
next  day  "but  was  interrupted  and  before  he  could  resume  the 
end  of  the  period  had  come  and  the  students  were  leaving  the 
room. 
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Sxhibit  3 


PRACTICAL  USES  FOR  A  COLLEGE 
TEACK5RS  RATIUa  SCALE 

Item  13,  page  192,  suggests  the  desirability  of 
further  comment  on  the  possible  uses  of  the  College  Teachers 
Rating-  Scale.     The  purpose  of  tracher  rating  is  the  iraproveraent 
of  college  instruction.     Interfering  traditions  have  ma^de  most 
difficult  any  approach  to  the  improvement  of  college  and 
university  teaching.    Does  the  College  Teachers  Rating  Scale 
offer  such  an  approach?      The  use  of  the  scale  at  lov/a  State 
College  suggests  that  it  does,  when  introduced  with  sufficient 
care  and  judgment. 

Because  of  the  existence  of  several  other  agencies  and 
activities  designed  to  increase  the  effectiveness  of  instruction 
on  the  Io7/a  State  College  cempus,  any  improvement  v/hich  may  have 
come  about  within  the  period  under  discussion,  cannot  be  attribu- 
ted solely  to  the  use  of  the  rating  scale.      Moreover,  it  would 
appear  impossible  to  attribute,  with  any  degree  of  accuracy,  to 
each  of  these  different  agencies  any  definite  proportion  of  the 
credit  for  v/hatever  improvement  in  teaching  which  may  have 
occurred  during  the  period  covered  by  this  investigation.  How- 
ever, it  may  not  be  inappropriate  to  note  some  of  the  changes 
which  have  come  about  at  the  Iowa  State  College  and  to  suggest 
that  so'^^.c  sha.re  of  the  improvement  should  be  credited  to  the 
use  of  the  College  Teachers  Rs.ting  Scale. 

Among  the  various  desirable  outcomes,  referred  to  above, 
the  following  are  probably  ar^ono:  the  most  import rjit: 
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1.  A  keen  and  active  interesl:  on  the  part  of  the 
members  of  the  faculty  in  the  improvement  of  their  teaching 
techniques, 

2.  An  increased  appreciation  of  the  importance  of 
methods  of  teaching,  and  of  the  personality  of  the  instructor, 
on  the  part  of  the  administrative  officers  who,  quite  generally, 
had  previously  regarded  research  work  as  ell-ioiportant . 

3.  The  stimijilation  of  experimentation  by  certain 
members  of  the  faculty/  on  problems  concerning  the  effective 
teaching  of  their  respective  subjects. 

4.  A  more  enlightened  and  critical  analysis  by  instruc- 
tors and  administrators,  of  the  important  abilities  and  qualities 
demanc^ed  of  a  successful  college  teacher. 

5.  A  greater  willingness  and  desire  on  the  part  of  the 
faculty  to  t.'^lk  over  their  teaching  problems  with  members  of  the 
education  department, 

6.  The  recognition  of  teaching  ability  of  a  high  grade 
by  promotion  in  rank  and  increase  in  selD.Ty, 

7.  A  more  critical  attitude,  on  the  part  of  the 
students,  concerning  the  quality  of  the  instruction  to  which 
they  are  subjected, 

3.    More  sympathetic  relationship  between  faculty  and 
students  as  a  result  of  the  increased  appreciation  by  the 
students  of  the  difficult  and  complex  nature  of  the  teaching 
task,  on  the  one  hand  and  on  the  other  hand  the  awakened 
interest,  by  the  fac^jlty,  in  the  student  as  an  individual  poe- 
essing  certain  rights  and  needs. 


9,    Ab  a  result  of  all  the  above,  a  significant  im- 
provement in  the  quality  of  the  instruction  offered  by  Iowa  State 
Oollec'e  during-  the  period  covered  ^Dy  this  investigstion , 

INTRODUOIKG  THE  USE  OF  THE  RATING  SCALE 

Any  college  faculty,  department  faculty,  or  individual 
instructor  who  may  be  interested  in  the  improvement  of  instruc- 
tion may  use  the  Rating  Scale  as  a  method  of  approach.  Students 
readily  aporecia^te  the  problem,  and  may  be  relied  upon  to  co- 
operate.    The  instructor  seeking  improvement  should  preface  the 
introduction  of  the  Rp.tinp;  Scale  by  re-'din.-:  some  j^'ood  books 
or  articles  on  the  improvement  of  instruction.    A  selected  list 
of  these  will  be  found  listed  in  the  bibliographer  ivhich  accompanies 
this  dissertation. 
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